Embedded pupil function recovery for fourier ptychographic imaging devices

ABSTRACT

Certain aspects pertain to Fourier ptychographic imaging systems, devices, and methods that implement an embedded pupil function recovery.

CROSS-REFERENCES TO RELATED APPLICATIONS

This application is a continuation-in-part application of U.S. patentapplication Ser. No. 14/065,280, titled “FOURIER PTYCHOGRAPHIC IMAGINGSYSTEMS, DEVICES, AND METHODS,” filed on Oct. 28, 2013, which claimspriority to U.S. Provisional Patent Application No. 61/720,258 entitled“Breaking the Spatial Product Barrier via Non-InterferometricAperture-Synthesizing Microscopy (NAM),” filed on Oct. 30, 2012 and toU.S. Provisional Patent Application No. 61/847,472 entitled “FourierPtychographic Microscopy,” filed on Jul. 17, 2013. This application isalso a continuation-in-part application of U.S. patent application Ser.No. 14/466,481 titled “VARIABLE-ILLUMINATION FOURIER PTYCHOGRAPHICIMAGING DEVICES, SYSTEMS, AND METHODS,” filed on Aug. 22, 2014, whichclaims priority to U.S. Provisional Patent Application No. 61/899,715,titled “Increasing Numerical Aperture of Dry Objective to Unity viaFourier Ptychographic Microscopy” and filed on Nov. 4, 2013; U.S.Provisional Patent Application No. 61/868,967, titled “AlternativeOptical Implementations for Fourier Ptychographic Microscopy” and filedon Aug. 22, 2013; and U.S. Provisional Patent Application No.62/000,722, titled “Ultra-High NA Microscope via Fourier PtychographicMicroscopy” and filed on May 20, 2014. This application also claimsbenefit of U.S. Provisional Patent Application No. 61/968,833 titled“Sharp Focus Generation via EPRY-FPM and Adaptive Optics,” filed on Mar.21, 2014; U.S. Provisional Patent Application No. 61/916,981 titled“Embedded Pupil Function Recovery for Fourier Ptychographic Microscopy,”filed on Dec. 17, 2013, and U.S. Provisional Patent Application No.61/944,380 titled “Embedded Pupil Function Recovery for FourierPtychographic Microscopy,” filed on Feb. 25, 2014. These applicationsare hereby incorporated by reference in their entirety and for allpurposes.

FEDERALLY SPONSORED RESEARCH OR DEVELOPMENT

This invention was made with government support under Grant No. OD007307awarded by the National Institutes of Health. The government has certainrights in the invention.

BACKGROUND

Certain embodiments described herein generally relate to imagingtechniques. More specifically, certain aspects pertain to Fourierptychographic imaging methods that implement an embedded pupil functionrecovery technique.

Imaging lenses ranging from microscope objectives to satellite-basedcameras are physically limited in the total number of features they canresolve. These limitations are a function of the point-spread function(PSF) size of the imaging system and the inherent aberrations across itsimage plane field of view (FOV). Referred to as the space-bandwidthproduct, the physical limitation scales with the dimensions of the lensbut is usually on the order of 10 megapixels regardless of themagnification factor or numerical aperture (NA). A discussion ofspace-bandwidth product of conventional imaging systems can be found inLohmann, A. W., Dorsch, R. G., Mendlovic, D., Zalevsky, Z. & Ferreira,C., “Space—bandwidth product of optical signals and systems,” J. Opt.Soc. Am. A. 13, pages 470-473 (1996), which is hereby incorporated byreference for this discussion. While conventional imaging systems may beable to resolve up to 10 megapixels, there is typically a tradeoffbetween PSF and FOV. For example, certain conventional microscopeobjectives can offer a sharp PSF (e.g., 0.5 μm) across a narrow FOV(e.g., 1 mm), while others imaging systems with wide-angle lenses canoffer a wide FOV (e.g., 10 mm) at the expense of a blurry PSF (e.g., 5μm).

Certain interferometric synthetic aperture techniques that try toincrease spatial-bandwidth product are described in Di, J. et al., “Highresolution digital holographic microscopy with a wide field of viewbased on a synthetic aperture technique and use of linear CCD scanning,”Appl. Opt. 47, pp. 5654-5659 (2008); Hillman, T. R., Gutzler, T.,Alexandrov, S. A., and Sampson, D. D., “High-resolution, wide-fieldobject reconstruction with synthetic aperture Fourier holographicoptical microscopy,” Opt. Express 17, pp. 7873-7892 (2009); Granero, L.,Micó, V., Zalevsky, Z., and Garcia, J., “Synthetic aperturesuperresolved microscopy in digital lensless Fourier holography by timeand angular multiplexing of the object information,” Appl. Opt. 49, pp.845-857 (2010); Kim, M. et al., “High-speed synthetic aperturemicroscopy for live cell imaging,” Opt. Lett. 36, pp. 148-150 (2011);Turpin, T., Gesell, L., Lapides, J., and Price, C., “Theory of thesynthetic aperture microscope,” pp. 230-240; Schwarz, C. J., Kuznetsova,Y., and Brueck, S., “Imaging interferometric microscopy,” Optics letters28, pp. 1424-1426 (2003); Feng, P., Wen, X., and Lu, R.,“Long-working-distance synthetic aperture Fresnel off-axis digitalholography,” Optics Express 17, pp. 5473-5480 (2009); Mico, V.,Zalevsky, Z., Garcia-Martinez, P., and Garcia, J., “Synthetic aperturesuperresolution with multiple off-axis holograms,” JOSA A 23, pp.3162-3170 (2006); Yuan, C., Zhai, H., and Liu, H., “Angular multiplexingin pulsed digital holography for aperture synthesis,” Optics Letters 33,pp. 2356-2358 (2008); Mico, V., Zalevsky, Z., and Garcia, J., “Syntheticaperture microscopy using off-axis illumination and polarizationcoding,” Optics Communications, pp. 276, 209-217 (2007); Alexandrov, S.,and Sampson, D., “Spatial information transmission beyond a system'sdiffraction limit using optical spectral encoding of the spatialfrequency,” Journal of Optics A: Pure and Applied Optics 10, 025304(2008); Tippie, A. E., Kumar, A., and Fienup, J. R., “High-resolutionsynthetic-aperture digital holography with digital phase and pupilcorrection,” Opt. Express 19, pp. 12027-12038 (2011); Gutzler, T.,Hillman, T. R., Alexandrov, S. A., and Sampson, D. D., “Coherentaperture-synthesis, wide-field, high-resolution holographic microscopyof biological tissue,” Opt. Lett. 35, pp. 1136-1138 (2010); andAlexandrov, S. A., Hillman, T. R., Gutzler, T., and Sampson, D. D.,“Synthetic aperture Fourier holographic optical microscopy,” Phil.Trans. R. Soc. Lond. A 339, pp. 521-553 (1992), all of which are herebyincorporated by reference for the discussion of attempts to increasespatial bandwidth. Most of the above-described interferometric syntheticaperture techniques include setups that record both intensity and phaseinformation using interferometric holography such as off-line holographyand phase-shifting holography. Interferometric holography has itslimitations. For example, interferometric holography recordingstypically use highly coherent light sources. As such, the constructedimages typically suffer from coherent noise sources such as specklenoise, fixed pattern noise (induced by diffraction from dust particlesand other optical imperfections in the beam path), and multipleinterferences between different optical interfaces. Thus the imagequality is typically worse than from a conventional microscope. On theother hand, using off-axis holography sacrifices spatial-bandwidthproduct (i.e., reduces total pixel number) of the image sensor. Adiscussion of certain off-axis holography methods can be found inSchnars, U. and Jüptner, W. P. O., “Digital recording and numericalreconstruction of holograms,” Measurement Science and Technology, 13,R85 (2002), which is hereby incorporated by reference for thisdiscussion. In addition, interferometric imaging techniques may subjectto uncontrollable phase fluctuations between different measurements.Hence, accurate a priori knowledge of the sample location may be neededto set a reference point in the image recovery process. Anotherlimitation is that many of these interferometric imaging systems requiremechanical scanning to rotate the sample and thus precise opticalalignments, mechanical control at a sub-micron level, and associatedmaintenances are required by these systems. In terms ofspatial-bandwidth product, these interferometric imaging systems maypresent little to no advantage as compared with a conventionalmicroscope.

Previous lensless microscopy such as in-line holography andcontact-imaging microscopy also present drawbacks. For example,conventional in-line holography does not work well with contiguoussamples and contact-imaging microscopy requires a sample to be in closeproximity to the sensor. A discussion of certain digital in-lineholography devices can be found in Denis, L., Lorenz, D., Thiebaut, E.,Fournier, C. and Trede, D., “Inline hologram reconstruction withsparsity constraints,” Opt. Lett. 34, pp. 3475-3477 (2009); Xu, W.,Jericho, M., Meinertzhagen, I., and Kreuzer, H., “Digital in-lineholography for biological applications,” Proc. Natl Acad. Sci. USA 98,pp. 11301-11305 (2001); and Greenbaum, A. et al., “Increasedspace—bandwidth product in pixel super-resolved lensfree on-chipmicroscopy,” Sci. Rep. 3, page 1717 (2013), which are herebyincorporated by reference for this discussion. A discussion of certaincontact-imaging microscopy can be found in Zheng, G., Lee, S. A.,Antebi, Y., Elowitz, M. B. and Yang, C., “The ePetri dish, an on-chipcell imaging platform based on subpixel perspective sweeping microscopy(SPSM),” Proc. Natl Acad. Sci. USA 108, pp. 16889-16894 (2011); andZheng, G., Lee, S. A., Yang, S. & Yang, C., “Sub-pixel resolvingoptofluidic microscope for on-chip cell imaging,” Lab Chip 10, pages3125-3129 (2010), which are hereby incorporated by reference for thisdiscussion.

A high spatial-bandwidth product is very desirable in microscopy forbiomedical applications such as pathology, haematology, phytotomy,immunohistochemistry, and neuroanatomy. For example, there is a strongneed in biomedicine and neuroscience to image large numbers of histologyslides for evaluation. This need has prompted the development ofsophisticated mechanical scanning and lensless microscopy systems. Thesesystems increase spatial-bandwidth product using complex mechanisms withhigh precision to control actuation, optical alignment, and motiontracking. These complex mechanisms tend to be expensive to fabricate anddifficult to use and maintain.

BRIEF SUMMARY

Certain embodiments described herein generally relate to imagingtechniques. More specifically, certain aspects pertain to Fourierptychographic imaging systems, devices, and methods that can be used inhigh resolution imaging applications such as, for example, pathology,haematology, semiconductor wafer inspection, and X-ray and electronimaging.

Certain embodiments pertain to a Fourier ptychographic imaging systememploying embedded pupil function recovery. The Fourier ptychographicimaging system comprising a variable illuminator configured toilluminate a sample at a plurality of oblique illumination incidenceangles, an objective lens configured to filter light issuing from thesample based on its numerical aperture, and a radiation detectorconfigured to receive light filtered by the lens and capture a pluralityof intensity images corresponding to the plurality of obliqueillumination incidence angles. The Fourier ptychographic imaging systemfurther comprises a processor configured to iteratively andsimultaneously update a pupil function and a separate sample spectrum.The sample spectrum is updated iteratively for each illuminationincidence angle at overlapping regions in the Fourier domain withFourier transformed intensity image data. The overlapping regionscorrespond to the plurality of illumination incidence angles and thenumerical aperture of the objective lens. In some cases, the processoris further configured to inverse transform the updated sample spectrumto determine an image of the sample, wherein the image has a higherresolution than the captured intensity images. In an adaptive opticsembodiment, the Fourier ptychographic imaging system further comprises awavefront modulator. In this case, the processor is further configuredto determine an aberration from the updated pupil function and thewavefront modulator is configured to adaptively correct an incidentwavefront based on the determined aberration.

Certain embodiments pertain to a Fourier ptychographic imaging methodthat employs embedded pupil function recovery. This method comprisesilluminating a sample from a plurality of incidence angles using avariable illuminator and filtering light issuing from the sample usingan optical element. The method further comprises capturing a pluralityof variably-illuminated intensity image distributions of the sampleusing a radiation detector. In addition, the method simultaneouslyupdates a pupil function and a separate sample spectrum. The samplespectrum is updated in overlapping regions with Fourier transformedvariably-illuminated intensity images measurements. The overlappingregions correspond to the plurality of incidence angles and thenumerical aperture of the lens. The method further comprises inverseFourier transforming the recovered sample spectrum to recover an imagehaving a higher resolution than the intensity images. In an adaptiveoptics embodiment, the Fourier ptychographic imaging method furthercomprises determining an aberration from the updated pupil function andadaptively correcting for the determined aberration using a wavefrontmodulator.

These and other features are described in more detail below withreference to the associated drawings.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a block diagram of components of a Fourier ptychographicimaging system, according to embodiments.

FIG. 2 depicts a schematic diagram of a side view of components of aFourier ptychographic imaging device in trans-illumination mode,according to embodiments.

FIG. 3 is a schematic diagram of a Fourier ptychographic imaging devicecomprising a variable illuminator in the form of a two-dimensional(10×10) matrix of 100 light elements, according to an embodiment.

FIG. 4 is a photograph of a Fourier ptychographic imaging system withcomponents in modular form, according to an embodiment.

FIG. 5 is a photograph of one of the light elements of the variableilluminator the Fourier ptychographic imaging device of FIG. 4.

FIG. 6 is a schematic diagram of a side view of components of a Fourierptychographic imaging device, according to embodiments.

FIG. 7 is a schematic diagram of a side view of components of a Fourierptychographic imaging device, according to embodiments.

FIG. 8 is a schematic diagram of a side view of components of a Fourierptychographic imaging device, according to embodiments.

FIG. 9 depicts an orthogonal view of components of a high NA Fourierptychographic imaging device with a circular variable illuminator.

FIGS. 10A and 10B depict an expansion in the Fourier domain for the highNA configuration shown in FIG. 9.

FIGS. 11A and 11B depict an expansion in the Fourier domain for a highNA configuration shown in FIG. 9 modified with a circular variableilluminator having two concentric rings.

FIG. 12 depicts an orthogonal view of components of a high NA Fourierptychographic imaging device with a rectangular array variableilluminator.

FIG. 13 depicts an orthogonal view of components of a Fourierptychographic imaging device in epi-illumination mode.

FIG. 14 depicts an orthogonal view of components of a Fourierptychographic imaging device in epi-illumination mode.

FIG. 15 depicts an orthogonal view of components of a Fourierptychographic imaging device in epi-illumination mode.

FIG. 16 depicts an orthogonal view of components of a Fourierptychographic imaging device in epi-illumination mode

FIG. 17 is a flowchart of a Fourier ptychographic imaging method.

FIG. 18 is a flowchart of an example of certain sub-steps of one of thesteps of the method of FIG. 17.

FIG. 19 is a flowchart of an example of certain sub-steps of one of thesteps of the method of FIG. 17.

FIGS. 20A and 20B are schematic illustrations depicting components of aFourier ptychographic imaging device in trans-illumination mode.

FIG. 21 is an illustration of certain steps of the Fourier ptychographicimaging method described with reference to FIGS. 10 and 12A.

FIG. 22 is a flowchart of a Fourier ptychographic imaging method withtile imaging.

FIG. 23 is a flowchart depicting details of a Fourier ptychographicimaging method that implement an Embedded Pupil Function Recovery (EPRY)technique, according to an embodiment.

FIG. 24 illustrates nine images resulting recovered from a Fourierptychographic imaging method using aberration correction with andwithout the EPRY technique, according to embodiments.

FIG. 25 is a plot of mean square error E² (b) vs. iterations toconvergence for runs of a Fourier ptychographic imaging method with andwithout the EPRY technique, according to embodiments.

FIG. 26 depicts six (6) images of reconstructed image data recoveredfrom a Fourier ptychographic imaging method without aberrationcorrection and a Fourier ptychographic imaging method implementing theEPRY technique, according to embodiments.

FIG. 27 is a plot of an example of a decomposition of different types ofaberration from low order to high order according to the mode number,according to an embodiment.

FIG. 28 depicts reconstructed sample image and wavefront aberrationimages at different regions over the field of view (FOV) from a run of aFourier ptychographic imaging method implementing the EPRY technique,according to the embodiments.

FIG. 29 is a wide FOV color image of a pathology slide, according to anembodiment.

FIG. 30 depicts the reconstructed sample image and wavefront aberrationimages from three different regions of the FOV color image of FIG. 29over the field of view from a run of a Fourier ptychographic imagingmethod implementing the EPRY technique, according to an embodiment.

FIG. 31 depicts reconstructed images of a USAF target at differentfields of view recovered from a Fourier ptychographic imaging methodwithout aberration correction, from a Fourier ptychographic imagingmethod with a pre-characterized aberration correction, and from aFourier ptychographic imaging method implementing EPRY technique withouta pre-characterized aberration, according to embodiments.

FIG. 32 is a side view of a Fourier ptychographic imaging system thatuses adaptive optics to correct for an aberration determined from apupil function recovered by the EPRY technique, according to anembodiment.

FIG. 33 is a side view of a Fourier ptychographic imaging system thatuses adaptive optics to correct for an aberration determined from apupil function recovered by the EPRY technique, according to anembodiment.

FIG. 34 is a block diagram of subsystems that may be present in aFourier ptychographic imaging system.

DETAILED DESCRIPTION

Certain embodiments described herein pertain to Fourier ptychographicimaging systems, devices, and methods.

I. Fourier Ptychographic Imaging Systems

In certain embodiments, a Fourier ptychographic imaging system comprisesa variable illuminator, an optical system, and a radiation detector. Insome cases, the Fourier ptychographic imaging system may be incommunication with a processor or further comprise a processor (e.g.,microprocessor). The variable illuminator can illuminate (e.g., withplane wave illumination) a sample being imaged from a plurality ofincidence angles at different sample times. The optical system canreceive light issuing from the sample and propagate it to the radiationdetector. The optical system comprises at least one filtering opticalelement that can “filter” light typically based on its acceptance angle.The radiation detector receives light from the optical system, andmeasures a light intensity distribution for each of the incidence anglesto capture a plurality of intensity images of the sample correspondingto the different incidence angles. The image data for each intensityimage is associated with a region in Fourier space. In the case of afiltering optical element in the form of a lens, the diameter of theregion in Fourier space corresponds to the NA of the lens and the centerof the region corresponds to the incidence angle of the illumination atthat sample time. In certain aspects, components of the Fourierptychographic imaging system (e.g., variable illuminator and filteringoptical element) are configured to acquire light intensity distributionsin the spatial domain that correspond to overlapping circular regions inthe Fourier space. In some cases, the components and their incidenceangles are designed to overlap the regions in Fourier space by apredefined amount and/or so that the overlapping regions cover apredefined area (e.g., an area that covers higher frequencies). Forexample, the NA of the filtering optical element and the number andlocations of discrete light elements of a variable illuminator may bedesigned so that circular pupil regions in Fourier space overlap by apredefined amount. In one case, components may be designed so that thecircular regions associated with adjacent incident angles overlap by apredefined percentage such as about 60%, about 70%, about 80%, about90%, etc. in Fourier space. The processor may be configured toiteratively stitch together the overlapping image data in Fourier space.The overlapping image data in Fourier space can be used to generate ahigher resolution image of the sample. In some cases, the Fourierptychographic imaging system can correct for aberrations in the system.In some cases, the Fourier ptychographic imaging system can refocus thehigher-resolution image.

The optical system comprises one or more components configured tocollect light issuing from the sample and propagate it to the radiationdetector. For example, the optical system may comprise a collectionoptical element configured to collect light issued from the sample. Asanother example, the optical system may comprise a filtering opticalelement configured to filter incident light. In one case, the filteringoptical element is in the form of an objective lens, which filters lightby rejecting incident light outside its acceptance angle and acceptinglight within its acceptance angle. In some cases, the collection opticalelement may also function as the filtering optical element. The opticalsystem propagates the filtered light to the radiation detector, whichmeasures (e.g., records) an intensity distribution at the radiationdetector at M sample times, t_(q=1 to M), to capture a plurality of Mintensity images of the sample. In certain cases, M=N, i.e. an intensitymeasurement corresponds to each incidence angle.

In some embodiments, Fourier ptychographic imaging system comprises anoptical system having a filtering optical element in the form of arelatively low NA objective lens (e.g., 2× lens with 0.08). This low NAsystem has a wide field-of-view (e.g., 13 mm in diameter) of the sample.In these cases, the system acquires intensity images with relatively lowresolution due to the low NA optical element filtering light issuingfrom the sample. These intensity images correspond to smaller circularregions in Fourier space than if a higher NA optical element were used.In order to overlap these smaller circular regions in Fourier space by acertain amount (e.g., 70%, 75%, etc.), the variable illuminator in thissystem is configured to provide illumination with relatively shortspacing (e.g., 0.05 rad) between adjacent incidence angles. Examples ofa Fourier ptychographic system with a low NA filtering optical elementfor wide field-of-view imaging can be found in U.S. patent applicationSer. No. 14/065,280, titled “Fourier Ptychographic Imaging Systems,Devices, and Methods” and filed on Oct. 28, 2013 and in U.S. patentapplication Ser. No. 14/065,305, titled “Fourier Ptychographic X-rayImaging Systems, Devices, and Methods,” and in G. Zheng, R. Horstmeyerand C. Yang, “Wide-field, high-resolution Fourier ptychographicmicroscopy,” Nature Photonics, 2013, which are hereby incorporated byreference in their entirety.

In some embodiments, Fourier ptychographic imaging system comprises anoptical system having a filtering optical element with a relatively highNA (e.g., 20× lens with 0.5 NA) and/or a higher illumination NA toincrease the combined system NA. Intensity images captured by this highNA system correspond to larger regions in Fourier space than intensityimages captured with a low NA system. Since larger regions are covered,the variable illuminator can be configured with reduced spacing betweenadjacent incidence angles and with a reduced number N of incidenceangles. In these systems, fewer intensity images may be needed togenerate the same or higher resolution than with systems using a low NAfiltering optical element. Since fewer intensity images may be needed,the image acquisition time may be shorter and may require fewercomputing resources to generate an image with the same or higherresolution than the low NA system. Also, the variable illuminator can beof a simpler design (e.g., less dense LED matrix) since fewer lightelements are needed to provide illumination from the reduced number N ofincidence angles. In some cases, the variable illuminator may be furtherconfigured so that the difference between extreme incidence angles islarger (i.e., higher illumination NA) than with the low NA systemdescribed above. That is, a higher illumination NA allows for capturingof high frequency data at the outer regions in Fourier space which alsoimproves the resolution of the final images. Thus, the high NA systemwith an increased illumination NA and/or an increased optical system NAcan provide for an increased system NA that can improve resolution ofimages. This high NA system may be able to illuminate the sample withincidence angles that allow for acquisition of images that cover largeroverlapping regions in Fourier space and higher frequency data. Whencombined, these overlapping larger regions can result in a synthesizedlarge system NA region that may, in certain cases, be close to unity. Incertain cases, these high NA systems have a high synthesized system NA(e.g., close to unity where the intrinsic NA of the filtering lightelement is lower such as, for example, about 0.75) while maintaining alarge working distance, and without using needing an immersion medium.

In conventional microscopes, the highest system NA that can be achievedis limited by geometric principle (i.e. at most the entire upperhemisphere light cone of light from the sample is collected) and lensdesign technology, resulting in an upper bound of ˜0.95 for drymicroscope and ˜1.40 for oil immersion microscope. Some conventionalwater or oil immersion objectives may provide NA>0.9 where an immersionmedia with refractive index greater than 1 improves collection of lightfrom the sample. However, immersion objectives have several drawbacksthat may make them unsuitable for some applications. Firstly, samplesneed to be immersed in media and typically the working distance is veryshort (0.1˜0.2 mm), which presents an obstacle for micromanipulation ofthe sample. Secondly, common immersion media have inherently highabsorption characteristics in the ultraviolet region (<375 nm) and nearinfrared region (>700 nm) of the spectrum, which brings some problem tothe bright-field immersion microscopy in this region and alsofluorescence immersion microscopy. A description of the relationshipbetween oil immersion and numerical aperture can be found at:http://www.olympusmicro.com/primer/anatomy/immersion.html, which ishereby incorporated by reference for this description.

In some embodiments described herein, the Fourier ptychographic imagingsystem has components configured to operate in trans-illumination modeso that illumination is directing through the sample and toward thecollection optics. In a Fourier ptychographic imaging system configuredto operate in trans-illumination mode, reflected light may not becaptured by the collection optical element and it may be that only lighttransmitted through the sample is collected.

In some embodiments described herein, the Fourier ptychographic imagingsystem has components configured are configured so that the collectionoptical elements receive reflected light from the surface of the sample.In a system configured to operate in epi-illumination mode, thecomponents are configured so that illumination is directed toward sampleand away from collection optical element. In such as configured system,the illumination source is configured to direct illumination to thesample from the same side as where the collection optical element islocated. Some examples of Fourier ptychographic imaging devices shownoperating in the epi-illumination mode are shown in FIGS. 13, 14, 15,and 16. A system configured to operate in epi-illumination mode may bemore effective for imaging thick and/or non-transparent samples than atrans-illumination mode. The Fourier ptychographic imaging systemsoperating in epi-illumination mode typically image reflective surfacesof the sample. Configuring Fourier ptychographic imaging systems forepi-illumination mode may be particularly useful in applications thatinvolve metal or semiconductor surface inspection including, forexample, semiconductor wafer, chip, and/or electronic circuit boardinspection, among others. Some applications for these Fourierptychographic imaging systems configured for epi-illumination mode mayinclude hand-held cameras with a modified flash system or satelliteimagery.

FIG. 1 is a block diagram of components of a Fourier ptychographicimaging system 10, according to certain embodiments. The Fourierptychographic imaging system 10 comprises a Fourier ptychographicimaging device 100 and a computing device 200 in electroniccommunication with the Fourier ptychographic imaging device 100. Incertain illustrated examples, such as the one shown in FIG. 1, a sampleis shown provided to the Fourier ptychographic imaging device during animage measurement (acquisition) process. For example, in FIG. 1, theFourier ptychographic imaging device 100 comprises an optional (denotedby dashed line) sample 20 that is present during an image measurement(acquisition) process. It will be understood that a sample in not anessential component of the device, and is being shown for the purposesof illustrating an operation of the device. The computing device 200 canbe in various forms such as, for example, a smartphone, laptop, desktop,tablet, etc. Various forms of computing devices would be contemplated byone skilled in the art. Although the computing device 200 is shown as acomponent separate from Fourier ptychographic imaging device 100, thetwo components may be in the same housing and/or may sharesub-components.

In FIG. 1, the Fourier ptychographic imaging device 100 comprises avariable illuminator 110, an optical system 130, and a radiationdetector 140. A variable illuminator is configured to provideillumination at a plurality of N incidence angles at (θx_(i,j),θy_(i,j)), i=1 to n, j=1 to m to a sample being imaged. In some cases,the variable illuminator is configured to illuminate the sample in atrans-illumination mode. In other cases, the variable illuminator isconfigured to illuminate the sample in an epi-illumination mode. In thetrans-illumination mode, the variable illuminator directs illuminationthrough the sample and toward a collection optical element of theoptical system. In an epi-illumination mode, the variable illuminatordirects illumination to the sample and away from a collection opticalelement of the optical system.

In FIG. 1, the computing device 200 comprises a processor 210 (e.g., amicroprocessor), a computer readable medium (CRM) 220 in communicationwith the processor 210, and a display 230 also in communication with theprocessor 210. The processor 210 is in electronic communication with theradiation detector 140 to receive signal(s) with image datacorresponding to M intensity images. The image data may include, forexample, intensity distributions, associated acquisition times, etc.

The processor 210 is in electronic communication with CRM 220 (e.g.,memory) to be able to transmit signals with image data in order to storeto and retrieve image data from the CRM 220. Processor 210 is inelectronic communication with display 230 to be able to send image dataand instructions to display images and other output, for example, to auser of the system 10. As shown by a dotted line, the variableilluminator 110 may optionally be in electronic communication withprocessor 210 to send instructions for controlling variable illuminator110. For example, in certain aspects these control instructions may beimplemented to synchronize the illumination times at different incidenceangles with the sample times of the radiation detector 140. Theelectronic communication between components of system 10 and othersystems and devices described herein may be in wired or wireless form.

The processor 210 may also receive instructions stored on the CRM 220and execute those instructions to perform one or more functions ofFourier ptychographic imaging system 10. For example, the processor 210may execute instructions to perform one or more steps of the Fourierptychographic imaging method. As another example, the processor 210 mayexecute instructions for illuminating light elements of the variableilluminator 110. As another example, the processor 210 may executeinstructions stored on the CRM 220 to perform one or more otherfunctions of the system such as, for example, 1) interpreting image datafrom the plurality of intensity images, 2) generating a higherresolution image from the image data, and 3) displaying one or moreimages or other output from the Fourier ptychographic imaging method onthe display 230.

The CRM (e.g., memory) 220 can store instructions for performing certainfunctions of the system 10. These instructions are executable by theprocessor 220 or other processing components of the system 10. The CRM220 can also store the (lower resolution) intensity and higherresolution image data, and other data produced by the system 10.

The Fourier ptychographic imaging system 10 also includes a display 230in electronic communication with the processor 210 to receive data(e.g., image data) and provide display data to the display 230 for, forexample, an operator of the Fourier ptychographic imaging system 10. Thedisplay 230 may be a color display or a black and white display. Inaddition, the display 230 may be a two-dimensional display or athree-dimensional display. In one embodiment, the display 230 may becapable of displaying multiple views.

In one operation, the Fourier ptychographic imaging system 10 of FIG. 1performs a Fourier ptychographic imaging method. This method comprises ameasurement (acquisition) process, a recovery process, and an optionaldisplay process. During the measurement process, the sample isilluminated from a plurality of N incidence angles (θx_(i,j), θy_(i,j)),i=1 to n, j=1 to m, (N=n×m) using the variable illuminator 110. Theoptical system 130 has a filtering optical element that filters lightissuing from the sample. The optical system 130 propagates the filteredlight to the radiation detector 140. The radiation detector 140 receivesthe filtered light and acquires a plurality of M intensity images,k_(k,l), k=1 to o and j=1 to p, where M=o×p. In certain cases, M may beN. The variable illuminator 110 is configured to generate illuminationat incidence angles that will generate image data in Fourier space thatoverlaps by a certain amount. During the recovery process, the Mintensity images are iteratively combined in Fourier space to generate ahigher-resolution image data (intensity and/or phase). During theoptional display process, an image (e.g., higher-resolution image,acquired intensity image, etc.) and/or other output may be provided on adisplay 230. In certain aspects, the system 10 may also be able tocorrect for any aberrations in the system 10, including re-focusing ofthe higher-resolution image. In one case, the system 10 may also be ableto propagate the higher resolution image to one or more planes. Theimage data from these propagated images at different planes can be usedto generate a three-dimensional image. In certain aspects, the system 10may also be able to generate an image at different illuminationwavelengths (RGB) to generate a color image.

Certain modifications, additions, or omissions may be made to theFourier ptychographic imaging system 10 without departing from the scopeof the disclosure. In addition, the components of the Fourierptychographic imaging system 10 or the components of the Fourierptychographic imaging devices described herein may be integrated orseparated according to particular needs. For example, the computingdevice 200 or components thereof may be integrated into the Fourierptychographic imaging device 100. In some embodiments, the processor 210or other suitable processor may be part of the Fourier ptychographicimaging device 100. In some cases, the processor 210 may be integratedinto a radiation detector so that the radiation detector performs thefunctions of the processor 210. As another example, the CRM 220 and/ordisplay 230 may be omitted from the Fourier ptychographic imaging system100 in certain cases.

In certain aspects, the Fourier ptychographic imaging systems anddevices may further comprise a receptacle for receiving the sample at asample surface. The sample surface may be part of a component of or aseparate component of the systems and devices.

In certain aspects, the field-of-view of the collection components ofthe Fourier ptychographic imaging system 10 may be divided into one ormore tile images. In these cases, the processor may construct a higherresolution complex image for each tile independently, and then combinethe tile images to generate a full field-of-view image. This ability toprocess tile images independently allows for parallel computing. Inthese aspects, each tile may be represented by a two-dimensional area.In polar spatial coordinates, each tile may be a circular area or anoval area. In rectilinear spatial coordinates, the full field-of viewlow resolution image may be divided up into a two-dimensional matrix oftiles in a rectangular area. In some embodiments, the dimensions of atwo-dimensional square matrix of tiles may be in powers of two whenexpressed in number of pixels of the radiation detector such as, forexample, a 256 by 256 matrix, a 64×64 matrix, etc.

A variable illuminator can refer to a device that is configured toprovide incident radiation to the sample being imaged at N differentincidence angles at M image acquisition times. In many cases, thevariable illuminator is designed to provide incident radiation at aplurality of N incidence angles (θx_(i,j), θy_(i,j)), i=1 to n, j=1 tom. Generally, N has a value in a range from 2 to 1000. In one case,N=100. In another case, N=200. Each incidence angle corresponds to alocation of the corresponding acquired image data in Fourier space.Adjacent incidence angles in the spatial domain correspond toneighboring regions in Fourier space. In certain embodiments, thevariable illuminator is designed to provide illumination at incidenceangles that provide for an overlapping area of neighboring regions ofimage data in Fourier space where the overlapping area is of at least acertain minimum amount (e.g. 75% overlap, 70% overlap, 80% overlap,etc.). To provide this minimum amount of overlap of neighboring regionsin Fourier space, the variable illuminator may be configured so that thedifference between adjacent incidence angles in the plurality of Nincidence angles is less than a certain maximum angular difference. Thatis, the variable illuminator may be configured with a maximum differencebetween adjacent incidence angles to provide the minimum amount ofoverlap in Fourier space. For example, the maximum angular differencemay be about 0.05 rad for a 2×0.08NA objective lens. In another case,the maximum angular difference may be about 0.1 rad.

The Fourier ptychographic imaging system comprises a filtering opticalelement. In some cases, the filtering optical element may comprise alens having an acceptance angle. This acceptance angle is associatedwith the diameter of a circular pupil region in Fourier space.

In some of these cases, the variable illuminator may be configured tohave adjacent incidence angles that are separated by an angle of a valuedefined by the acceptance angle of the lens. In one case, the value ofthe difference between two adjacent incidence angles of the plurality ofincidence angles may be in the range of about 10% to about 90% of theacceptance angle of the filtering optical element in the form of anobjective lens. In another case, the value of the difference between twoadjacent incidence angles of the plurality of incidence angles may be inthe range of 33% and 66% of the acceptance angle of the filteringoptical element in the form of an objective lens. In another case, thevalue of the difference between two adjacent incidence angles of theplurality of incidence angles may be less than about 76% of theacceptance angle of the filtering optical element in the form of anobjective lens. In another case, the difference between adjacentincidence angles is about ⅓ of the acceptance angle defined thefiltering optical element in the form of an objective lens. In anothercase, the range of incidence angles, defined by a difference between thelargest and smallest incidence angles, may be about equal to thenumerical aperture consistent with the spatial resolution of the finalhigher-resolution image. In one case, the acceptance angle is in therange of about −0.08 rad to about 0.08 rad, and the adjacent angle is0.05 rad.

In certain embodiments, the variable illuminator comprises one or moreradiation sources. Although the radiation source(s) are usually coherentradiation sources, incoherent radiation source(s) may also be used insome cases and computational corrections may be applied. The radiationsources may be visible light other forms of radiation. In cases that usevisible light radiation, the radiation source(s) is a visible lightsource. Some examples of a radiation source of visible light include aliquid crystal display (LCD) pixel and a pixel of a light emitting diode(LED) display. In cases that use other forms of radiation, other sourcesof radiation may be used. For example, in embodiments that use X-rayradiation, the radiation source may comprise an X-ray tube and a metaltarget. As another example, in cases that use microwave radiation, theradiation source may comprise a vacuum tube. As another example, inembodiments that use acoustic radiation, the radiation source may be anacoustic actuator. As another example, in embodiments that use Terahertzradiation, the radiation source may be a Gunn diode. One skilled in theart would contemplate other sources of radiation. In one case that usesTerahertz radiation, the frequencies of the radiation provided by theillumination source may be in the range of about 0.3 to about 3 THz. Inone case that uses microwave radiation, the frequencies of the radiationprovided by the variable illuminator may be in the range of about 100MHz to about 300 GHz. In one case that uses X-ray radiation, thewavelengths of the radiation provided by the variable illuminator may bein the range of about 0.01 nm to about 10 nm. In one case that usesacoustic radiation, the frequencies of the radiation provided by thevariable illuminator may be in the range of about 10 Hz to about 100MHz.

In certain cases, the variable illuminator may comprise a plurality ofdiscrete light elements, each light element comprising at least oneradiation source. For example, a variable illuminator that is configuredto provide visible light typically includes a plurality of discretelight elements. Some examples of discrete light elements that canprovide visible light are an LCD pixel and a pixel of an LED display. Inmany cases, the illumination provided by each light element may beapproximated as plane wave illumination at the sample from a singleincidence angle. For example, the light element 112(a) in FIG. 2provides illumination 114(a) at an incidence angle that has a componentθx_(i,j) in the x-z plane.

In certain cases, the properties (e.g., wavelength, frequency, phase,amplitude, polarity, etc.) of illumination from the activated radiationsource(s) of the variable illuminator at each acquisition time may beapproximately uniform. In some cases, the illumination from theactivated radiation source(s) at all acquisition times from allincidence angles may be approximately uniform. In other cases, theproperties may vary at the different incidence angles, for example, byproviding n different wavelengths λ₁, . . . , λ_(n) during themeasurement process. In other cases, the variable illuminator mayprovide RGB illumination of three wavelengths λ₁, λ₂, and λ₃corresponding to red, green, blue colors, respectively. In examples thatuse Terahertz radiation, the frequencies of the radiation provided bythe variable illuminator may be in the range of about 0.3 to about 3THz. In examples that use microwave radiation, the frequencies of theradiation provided by the variable illuminator may be in the range ofabout 100 MHz to about 300 GHz. In examples that use X-ray radiation,the wavelengths of the radiation provided by the variable illuminatormay be in the range of about 0.01 nm to about 10 nm. In examples thatuse acoustic radiation, the frequencies of the radiation provided by thevariable illuminator may be in the range of about 10 Hz to about 100MHz.

In some cases, the variable illuminator comprises a plurality of Nstationary discrete light elements at different spatial locations (e.g.,variable illuminator 110(b) in FIG. 3). These N stationary lightelements may illuminate, individually or in groups of one or more, atdifferent sample times (e.g., successively) to provide illumination fromthe plurality of N incidence angles. In other cases, the variableilluminator may comprise a moving light element. This moving lightelement may move relative to the optical system, the sample, and theradiation detector, which may be kept stationary. In these cases, themoving light element may be moved to a plurality of N different spatiallocations using a mechanism such as a raster scanner. Based on therelative movement between the stationary components and the moving lightelement, the light element can provide illumination from the pluralityof N incidence angles. In other cases, the variable illuminatorcomprises a stationary light element and the other components of systemare moved to different spatial locations to provide the relativemovement. Based on this relative movement between the stationary lightelement and the other components of the system, the light element canprovide illumination from the plurality of N incidence angles.

In cases having a variable illuminator comprising a plurality of lightelements, the light elements may be in various arrangements such as aline grid, a rectangular grid, one or more concentric circles (rings), ahexagonal grid, curvilinear grid, or other suitable arrangement capableof providing the illumination from the plurality of incidence angles. Anexample of a circular variable illuminator having light elements in theform a single ring is shown in FIG. 9. An example of rectangular arrayvariable illuminator in the form of a rectilinear grid of light elementsis shown in FIG. 12. Some examples of light elements are a pixel of aliquid crystal display (LCD) or a light emitting diode (LED). Thearrangement of light elements may be configured with a spacing betweenadjacent elements and at particular locations that when activated canprovide illumination at a plurality of incidence angles that correspondto overlapping regions in Fourier space, in some cases, with an overlapof a certain amount.

In cases with multiple light elements, the light elements locations maybe represented by a one-dimensional or two-dimensional array (e.g., 1×9array, 3×6 array, 10×10 array, 15×15 array, 32×32 array, 100×100 array,50×10 array, 20×60 array, or other array with two dimensions). In somecases, such a two-dimensional array has dimensions n×m with lightelement locations X_(i,j) (r, θ) or X_(i,j) (x, y), i=1 to n, j=1 to mwhere the number of locations, where N=n×m.

In certain aspects, the variable illuminator comprises discrete lightelements that are illuminated at different acquisition times in anorder, for example, according to illumination instructions. For example,the order may define the illumination times of individual light elementsor groups of light elements in a two-dimensional array of discrete lightelements. In one example where the two-dimensional matrix of lightelements is a rectangular array, a central light element may bedetermined. The illumination instructions may instruct to illuminate thecentral light element first, then illuminate the 8 light elementssurrounding the central light element going counterclockwise, thenilluminate the 16 light elements surrounding the previous light elementsgoing counterclockwise, and so on until the variable illuminator hasprovided illumination from the plurality of N incidence angles(θx_(i,j), θy_(i,j)), i=1 to N. In another example where thetwo-dimensional matrix of light elements is a polar matrix such as oneor more concentric rings, the illumination instructions may instruct toilluminate the light elements at smallest radius first (e.g., inclockwise, counterclockwise, or random order), then illuminate any lightelement at a larger radius, and so on until all the variable illuminatorhas provided illumination from the plurality of N incidence angles(θx_(i,j), θy_(i,j)), i=1 to N. In another example where thetwo-dimensional array of light elements is a rectangular or a polararray, a light element closest to the sample may be determined. Theillumination instructions may instruct to illuminate the light elementclosest to the sample, and then illuminate the light element nextclosest to the sample, and then illuminate the light element nextclosest, and so on until the N light elements have been illuminated fromthe plurality of N incidence angles. In another example, the lightelements may be illuminated in a random order. In another example, asequential column by column order may be followed such as, for example,(X₁,Y₁), (X₁,Y₂), (X₁,Y₃), . . . (X₂,Y₁), (X₁,Y₂), (X₁,Y₃), . . .(X₂,Y_(n)), . . . (X_(m),Y_(n)). Alternatively, a row by row order maybe followed.

In certain aspects, the variable illuminator may be configured tooperate in epi-illumination mode, in trans-illumination mode, or in bothepi-illumination mode and trans-illumination mode. To be able to operatein the epi-illumination mode, the variable illuminator is typicallylocated on the same side of the sample as the collecting optical elementof the optical system. To be able to operate in the trans-illuminationmode, the variable illuminator is typically located on the opposite sideof the sample as the collecting optical element of the optical system.

A sample being imaged by the Fourier ptychographic imaging systemsdescribed herein can be comprised of one or more objects and/or one ormore portions of an object. Each object may be, for example, abiological entity, an inorganic entity, etc. Some examples of biologicalentities that can be imaged include whole cells, cell components,microorganisms such as bacteria or viruses, and cell components such asproteins. An example of an inorganic entity that can be imaged is asemiconductor wafer. In certain aspects, a thick and/or non-transparentsample can be imaged by certain Fourier ptychographic imaging systemsdescribed herein. The sample may be provided in a medium such as aliquid.

In luminescence imaging examples, a reagent (e.g.,fluorescence/phosphorescence dye) may be mixed with the sample to markor tag portions under investigation with fluorophore. A fluorophore canrefer to a component of a molecule that causes the molecule to fluoresceor phosphoresce. A fluorophore can absorb energy from excitation lightof a specific wavelength(s) and re-emit the energy at a differentwavelength(s). In luminescence imaging examples, the illumination sourcemay illuminate the sample with excitation light of predeterminedwavelength(s) (e.g., blue light) to activate the fluorophore in thesample. In response, the fluorophore release emissions of a differentwavelength(s) (e.g., red light).

The optical system may comprise one or more other components such as,for example, lens(es), beam splitter(s), objective(s), tube lens(es),wavelength filter(s), aperture element(s) (e.g., objective, physicaliris, etc.), and other like elements. In a luminescence imaging example,the optical system may include, for example, a filter (e.g., materialthat passes emissions and blocks excitation light) between thecollection optics and the radiation detector to filter out excitationlight and pass emissions. The optical system may include, for example,certain microscope optical components or camera optical components.Generally, the optical system comprises a collection optical element orfirst optical element that collects light issuing from the sample. Theoptical system also comprises a filtering optical element for filteringlight issuing from the sample. The filtering optical element may be thecollection optical element. In certain cases, the filtering opticalelement may be a lens (e.g., an objective lens). In certain high NAexamples, the high NA of the lens may be about 0.50. In other high NAexamples, the high NA of the lens may be in the range of about 0.50 toabout 0.75. In another high NA example, the high NA of the lens may beabout 0.60.

In certain Fourier ptychographic imaging systems described herein, theradiation detector (e.g., radiation detector 140 in FIG. 1) isconfigured to acquire a plurality of intensity images of a sample bymeasuring/recording an intensity distribution of incident radiation at adetector plane at a particular sample (acquisition) time. During animage measurement process, for example, the radiation detector mayacquire a plurality of M intensity images at M sample times,t_(q=1 to M). If visible light radiation is being measured, theradiation detector may be in the form of a charge coupled device (CCD),a CMOS imaging sensor, an avalanche photo-diode (APD) array, aphoto-diode (PD) array, a photomultiplier tube (PMT) array, or likedevice. If using THz radiation, the radiation detector may be, forexample, an imaging bolometer. If using microwave radiation, theradiation detector may be, for example, an antenna. If X-ray radiationis used, the radiation detector may be, for example, an x-ray sensitiveCCD. If acoustic radiation is used, the radiation detector may be, forexample, a piezoelectric transducer array. These examples of radiationdetectors and others are commercially available. In some cases, theradiation detector may be a color detector e.g. an RGB detector. Inother cases, the radiation detector need not be a color detector. Incertain cases, the radiation detector may be a monochromatic detector.

In certain aspects, a Fourier ptychographic imaging system comprises avariable illuminator configured to illuminate the sample from aplurality of N illumination incidence angles and radiation detectorconfigured to capture a plurality of M intensity images based ondifferent incidence angles of the plurality of N incidence angles. Incertain cases, N=M (i.e. an intensity image is acquired for eachillumination angle).

In certain aspects, the radiation detector may have discrete lightdetecting elements (e.g., pixels). In some cases, the discrete lightdetecting elements may have a size in the range of 1-10 microns. In onecase, the discrete light detecting element may have a size of about 1micron. The discrete light detecting elements may be circular,rectangular (e.g., square), or the like. For example, a radiationdetector that is in the form of a CMOS or CCD array may havecorresponding CMOS or CCD elements that are 1-10 microns. In anotherexample, a radiation detector that is in the form of an APD or PMT arraymay have corresponding CMOS or CCD elements that are in the range of 1-4mm. In one example, the radiation detecting element is a square pixelhaving a size of 5.5 um.

A sample time or acquisition time can refer to a time that the radiationdetector captures an intensity image of the sample. During certain imagemeasurement processes described herein, the radiation detector capturesa plurality of M intensity images (e.g., M=1, 2, 5, 10, 20, 30, 50, 100,1000, 10000, etc.) at different sample/acquisition times. Typically, theradiation detector is configured so that the sampling rate is set tocapture an intensity image at different illumination incidence angles.In one example, the sampling rates may be in a range from 0.1 to 1000frames per second.

Fourier space may refer to a mathematical space spanned by wave vectorskx and ky being the coordinate space in which the two-dimensionalFourier transforms of the spatial images created by theaperture-scanning Fourier ptychographic imaging system reside. Fourierspace may also refer to the mathematical space spanned by wavevectors kxand ky in which the two-dimensional Fourier transforms of the spatialimages collected by the radiation sensor reside.

During the measurement (acquisition) process, the radiation detectorcaptures M images in the form of image data. In most cases, the capturedimage data at each sample time is a light intensity distributionmeasured by the discrete light detecting elements of the radiationdetector. That is, M intensity images are captured. In addition tointensity distribution data, the radiation detector may also generateother image data such as the sample times and other related sample data.The image data generated by the radiation detector may be communicatedto other components of the system such as the processor and/or display.

The image data for each of the M intensity images captured by theradiation detector is associated with a region in Fourier space. InFourier space, neighboring regions may share an overlapping area overwhich they sample the same Fourier domain data. The distance between theneighboring regions in Fourier space corresponds to the distance betweenneighboring incidence angles of illumination provided by the variableilluminator. In certain aspects, the variable illuminator may beconfigured to provide illumination at a plurality of incidence anglesthat provide a predefined amount of overlapping area betweencorresponding neighboring regions in the Fourier domain data. In onecase, the variable illuminator is configured to provide illumination ata plurality of incidence angles to generate an overlapping area in theFourier domain data in the range of about 2% to about 99.5% of the areaof one of the regions. In another case, the overlapping area betweenneighboring regions may have an area that is in the range of about 65%to about 75% the area of one of the regions. In another case, theoverlapping area between neighboring regions may have an area that isabout 65% of the area of one of the regions. In another case, theoverlapping area between neighboring regions may have an area that isabout 70% of the area of one of the regions. In another case, theoverlapping area between neighboring regions may have an area that isabout 75% of the area of one of the regions.

Based on the geometry of the components of the Fourier ptychographicimaging system, the variable illuminator may be configured to generateillumination from incidence angles that provide a predefined amount ofoverlapping area between neighboring regions in Fourier space. Forexample, there may be a predefined maximum distance between neighboringlight elements being illuminated at different acquisition times toprovide a minimum amount of overlap between neighboring regions inFourier space. In one case, the maximum distance between neighboringlight elements may be about 1 mm. In another case, the maximum distancebetween neighboring light elements may be about 0.5 mm. In another case,the maximum distance between neighboring light elements may be about 4mm.

In certain embodiments described herein, a Fourier ptychographic imagingsystem may be configured for luminescence (e.g., fluorescence,phosphorescence, chemluminescence, bioluminescence, etc.) imaging. Forexample, a Fourier ptychographic imaging system may be adapted tocollect emissions directed back toward the illumination source. Inluminescence imaging, fluorophores in the sample are excited byexcitation illumination of a certain wavelength(s) from the illuminationsource and emit light of a different wavelength(s) (emissions). Theseemissions tend to have a weak signal compared to the excitation light sothat collection efficiency may be important. In certain examples,Fourier ptychographic imaging system configured for luminescence imagingoperates in epi-illumination mode. By operating in epi-illuminationmode, the radiation detector can receive emissions from the sampleand/or light reflected from the sample back toward the illuminationsource. These examples have optical arrangements that can accommodate anillumination source that directs excitation illumination to the sampleand away from collection optical element of the system. With thisoptical arrangement, collection of excitation illumination may besubstantially avoided.

II. Fourier Ptychographic Imaging Device Configurations

Fourier ptychographic imaging devices may be configured for use withparticular types of radiation. For example, Fourier ptychographicimaging device 100(a) of FIGS. 2 and 100(b) of FIG. 3 may beparticularly suitable for use with visible light, Terahertz, and/ormicrowave radiation. As another example, Fourier ptychographic imagingdevice 100(e) of FIG. 7 may be particularly suitable for use with X-rayradiation.

FIG. 2 depicts a schematic diagram of a side view of components of aFourier ptychographic imaging device 100(a) of a trans-illuminationconfiguration according to embodiments. In FIG. 2, the Fourierptychographic imaging device 100(a) comprises a variable illuminator110(a), an optical system 130(a), and a radiation detector 140(a) havinga sensing surface 142(a). The variable illuminator 110(a) comprises alight element 112(a) and a surface 111(a). The variable illuminator110(a) also comprises an x′-axis, a y′-axis (not shown) at a planedepicting the approximated plane from which the source of illuminationis provided, and a z′-axis. Although radiation detector 140(a) is shownat a distance away from optical system 130(a), radiation detector 140(a)may optionally be located proximal optical system 130(a).

In the illustrated example, a sample 20(a) has been provided to aspecimen surface 126(a) for the measurement process. The light element112(a) is shown providing illumination 114(a) in a trans-illuminationmode through the sample 20(a) where the illumination 114(a) has awavevector kx_(i,j), ky_(i,j) for the measurement process. Also shown isan in-focus plane 122(a) at z=0 and a sample plane 124 at z=z₀. TheFourier ptychographic imaging device 100(a) further comprises an x-axis,a y-axis (not shown) at the in-focus plane 122, and a z-axis orthogonalto the in-focus plane 122. Also shown is a distance d between thevariable illuminator 110(a) and the sample plane 124 and a workingdistance d_(o) between the sample 20(a) and the optical system 130(a).Generally, a working distance, d₀, refers to the distance between thesample 20(a) and the collecting optical element of the optical system130(a).

In FIG. 2, the light element 112(a) is shown providing illumination114(a) at a single sample (acquisition) time in the measurement process.The optical system 130(a) receives and filters light issuing from sample20(a). Light filtered by the optical system 130(a) is received at thesensing surface 142(a) of the radiation detector 140(a). The radiationdetector 140(a) measures the intensity distribution of light incidentthe sensing surface 142(a) and captures an intensity image at the sampletime. Although the Fourier ptychographic imaging device 100(a) is shownat a single sample time, the device 100(a) may include N light elements112(a) illuminating at, for example, N incidence angles (θx_(i,j),θy_(i,j)), i=1 to n, j=1 to m, where N=n×m. In this case, the radiationdetector 140(a) may acquire a plurality of M intensity images I_(k,l),k=1 to o and j=1 top at the M sample times, where each intensity imagemay be acquired while the illumination is at a different incidence angleof the plurality of N incidence angles (θx_(i,j), θy_(i,j)). Theincidence angles (θx_(i,j), θy_(i,j)) are angles measured relative to anaxis normal to the sample plane at z=z₀ and through point P. In the sideview shown in FIG. 2, only the component θx_(i,j) of the incidence anglein the x-z plane is shown.

FIG. 3 depicts a schematic diagram of a side view of components of aFourier ptychographic imaging device 100(b), according to embodiments.Fourier ptychographic imaging device 100(b) comprises a variableilluminator 110(b) comprising a plurality of N stationary lightelements, arranged in a two-dimensional matrix format. In theillustrated example, the i^(th) light element 112(b) providesillumination from an incidence angle (θ_(x) ^(i), θ_(y) ^(i)). AlthoughFIG. 3 shows the variable illuminator 110(b) having a 10×10 matrix oflight elements 112, other dimensions can be used in other embodiments.Although FIG. 3 shows equally spaced light elements 112(b) other numbersand spacing may be used. Variable illuminator 110(b) also comprises anx′-axis, y′-axis (not shown), and a z′-axis. As shown, the stationarylight elements 112(b) extend in the x′-direction and the y′-direction.

Fourier ptychographic imaging device 100(b) further comprises an opticalelement 130(b) (e.g., objective lens) and a radiation detector 140(b)having a sensing surface 142. Although radiation detector 140(b) isshown at a distance away from optical element 130(b), radiation detector140(b) may optionally be located at the optical element 130(b). TheFourier ptychographic imaging device 100(b) also includes an in-focusplane 122 at z=0 and a sample plane 124 at z=z₀. The Fourierptychographic imaging device 100(b) includes an x-axis and a y-axis (notshown) at the in-focus plane 122, and a z-axis orthogonal to thein-focus plane 122. The Fourier ptychographic imaging device 100(b) alsoincludes a distance d between the variable illuminator 110(b) and thesample plane 124. In the illustrated example, specimen 20(b) is locatedat a specimen surface 126 for the acquisition process.

In FIG. 3, the Fourier ptychographic imaging device 100(b) is shown at aparticular sample time, t_(i), in the measurement process. At sampletime, t_(i), i^(th) light element 112 provides incident illumination ata wavevector associated with an incidence angle of (θ_(x) ^(i), θ_(y)^(i)). The optical element 130(b) receives and filters light issuingfrom specimen 20. Light filtered by the optical element 130(b) isreceived at the sensing surface 142 of the radiation detector 140(b).The radiation detector 140(b) senses the intensity distribution of thefiltered light and captures a low-resolution intensity image. AlthoughFourier ptychographic imaging device 100(b) is shown at a single sampletime, t_(i), the Fourier ptychographic imaging device 100(b) can operateat a plurality of N sample times, t_(i=1 to N), associated with Nincidence angles (θ_(x) ^(i), θ_(y) ^(i)), i=1 to N to capture Nlow-resolution two-dimensional intensity images.

In certain embodiments, components of a Fourier ptychographic imagingsystem may be placed in communication with components of a conventionalmicroscope or other conventional imaging device to transform theconventional device into a Fourier ptychographic imaging system. FIG. 4is a photograph of a Fourier ptychographic microscope system 11 that iscomprised of components of a conventional microscope, according to anembodiment. The Fourier ptychographic microscope system 11 comprises aFourier ptychographic microscope 100(c). The Fourier ptychographicmicroscope 100(a) comprises components of a Fourier ptychographicimaging system and components of an Olympus® BX 41 microscope totransform it into the Fourier ptychographic imaging system 11. Thecomponents of the Olympus® BX 41 microscope comprise a 2×, 0.08 NAobjective 130(c) that functions as the optical element. The field numberof the 2× objective lens is 26.5. The field-of-view of the Fourierptychographic microscope 100(c) at the sample plane is 13.25 mm indiameter. In this example, the components of a Fourier ptychographicimaging system include a programmable two-dimensional LED matrix 114 anda CCD camera (not shown) that has been placed under the specimen stage.The CCD camera functions as the radiation detector in this example. Thistwo-dimensional LED matrix 114 has been programmed to provide variableilluminations to function as the variable illuminator. The programmabletwo-dimensional LED matrix 114 comprises a plurality of LEDs. In thephotograph a single LED 112(c) is illuminated. FIG. 5 is a zoomed inphotograph of the illuminated LED 112(c). As shown in FIG. 5, the LED isconfigured with sub elements to provide red, green, and blueilluminations. Although not shown, a processor 210 may be in electroniccommunication with the two-dimensional LED matrix 114 and/or to CCDcamera through, for example, the wires 201.

In FIG. 4, a specimen 20(c) has been provided to the Fourierptychographic microscope 100(c) on a slide 202. During the acquisitionprocess, the red, green, and blue elements illuminate from the LEDs inthe two-dimensional LED matrix 114, and the CCD camera acquires red,green, and blue intensity images. From these intensity images, acomputing device can computationally reconstruct a high-resolution andwide field-of-view color image of the specimen area by iterativelycombining low-resolution measurements in Fourier space. In one case, theprocessor may computationally reconstruct high-resolution and widefield-of-view red, green, and blue images, and then combine the imagesto generate a color image.

In certain embodiments, a Fourier ptychographic imaging device furthercomprises a mechanism (e.g., scanning mechanism) for moving the lightelement or other components relative to the light element to generatevariable illumination. For example, the Fourier ptychographic imagingdevice 110(d) in FIG. 6 has a mechanism 150. As another example, theFourier ptychographic imaging device 110(f) in FIG. 7 has a mechanism160.

FIG. 6 is a schematic diagram of a side view of components of a Fourierptychographic imaging device 100(d), according to an embodiment. Fourierptychographic imaging device 100(d) comprises a variable illuminator110(d) comprising a light element 112(d) that is moved (e.g., scanned)in the x′-direction (direction on the x′-axis) and y′-direction(direction on the y′-axis) to a plurality of N locations. Variableilluminator 110(d) also comprises an x′-axis, y′-axis, and z′-axis. Inthe illustration, the light element 112(d) has moved from a normalincidence position (θ_(x) ^(i)=0, θ_(y) ^(i)=0) in the x′-direction to aposition that provides illumination at (θ_(x) ^(i)=−h, θ_(y) ^(i)=0).The light element 112(d) is moved using a mechanism 150 such as a rasterscanner.

Fourier ptychographic imaging device 100(d) further comprises an opticalelement 130(d) and a radiation detector 140(d) having a sensing surface142. Although radiation detector 140(d) is shown at a distance away fromoptical element 130(d), radiation detector 140(d) may optionally belocated at the optical element 130(d). The Fourier ptychographic imagingdevice 100(d) also includes an in-focus plane 122 at z=0 and a sampleplane 124 at z=z₀. The Fourier ptychographic imaging device 100(d)includes an x-axis and a y-axis (not shown) at the in-focus plane 122,and a z-axis orthogonal to the in-focus plane 122. The Fourierptychographic imaging device 100(d) also includes a distance d betweenthe variable illuminator 110(d) and the sample plane 124. In theillustrated example, specimen 20(d) has been located at a specimensurface 126 for imaging. In other embodiments, specimen 20(d) may be inother locations for imaging purposes.

In FIG. 6, the light element 112(d) is shown providing illumination atsample time, t_(i) in the measurement process. The optical element130(d) filters light it receives. Light filtered by the optical element130(d) is received at the sensing surface 142 of the radiation detector140(d). The radiation detector 140(d) senses the intensity distributionof the filtered light and captures a low-resolution intensity image ofthe specimen area. Although Fourier ptychographic imaging device 100(d)is shown at a single sample time, t_(i), the Fourier ptychographicimaging device 100(d) can operate at a plurality of N sample times,t_(i=1 to N), associated with N incidence angles (θ_(x) ^(i), θ_(y)^(i)), i=1 to N to capture N low-resolution two-dimensional intensityimages. In embodiments where the Fourier ptychographic imaging device100(d) shown in FIG. 6 is used with X-ray radiation, the light element112(d) includes an X-ray source.

FIG. 7 is a schematic diagram of a side view of components of a Fourierptychographic imaging device 100(e), according to an embodiment. Fourierptychographic imaging device 100(e) comprises a variable illuminator110(e) with a light element 112(e), an optical element 130(e), aradiation detector 140(e) having a sensing surface 142, and a mechanism160. In the illustrated example, specimen 20(e) has been provided to theFourier ptychographic imaging device 100(e) for imaging.

In FIG. 7, the mechanism 160 moves an assembly 170 comprising theoptical element 130(e), a radiation detector 140(b) and specimen 20(e)relative to the stationary light element 112(e) to provide illuminationfrom a plurality of N incidence angles. The mechanism 160 may translateand/or rotate the assembly 170. For example, the assembly 170 maymounted on a goniometer sate that would allow the assembly to be rotatedas a whole relative to the light element 112(e). The variableilluminator 110(e) also comprises an x′-axis, y′-axis, and z′-axis.

Although radiation detector 140(e) is shown at a distance away fromoptical element 130(e), radiation detector 140(e) may optionally belocated at the optical element 130(e). The Fourier ptychographic imagingdevice 100(e) also includes an in-focus plane 122 at z=0 and a sampleplane 124 at z=z₀. The Fourier ptychographic imaging device 100(e)includes an x-axis and a y-axis (not shown) at the in-focus plane 122,and a z-axis orthogonal to the in-focus plane 122. The Fourierptychographic imaging device 100(e) also includes a distance d betweenthe variable illuminator 110(e) and the sample plane 124.

In FIG. 7, the light element 112(e) is shown providing illumination atsample time, t_(i) in the measurement process. The optical element130(e) receives and filters light issuing from specimen 20(e). Lightfiltered by the optical element 130(e) is received at the sensingsurface 142 of the radiation detector 140(e). The radiation detector140(e) senses the intensity distribution of the filtered light andcaptures a low-resolution intensity image of the area. Although Fourierptychographic imaging device 100(e) is shown at a single sample time,t_(i), the Fourier ptychographic imaging device 100(e) can operate at aplurality of N sample times, t_(i=1 to N), associated with N incidenceangles (θ_(x) ^(i), θ_(y) ^(i)), i=1 to N to capture N low-resolutiontwo-dimensional intensity images.

FIG. 8 is a schematic diagram of a side view of components of an Fourierptychographic imaging device 100(f), according to embodiments. Fourierptychographic imaging device 100(f) comprises a variable illuminator110(f) with a light element 112(f) that is moved by rotating it, anoptical element 130(f), and a radiation detector 140(f) having a sensingsurface 142. Although not shown, a mechanism may also be included torotate the light element 112. In the illustrated example, specimen 20(f)has been provided to the Fourier ptychographic imaging device 100(f) forimaging. In some cases, the light element 112(f) may be a laser.

In FIG. 8, the light element 112(f) is moved by rotating it, whichprovides illumination at (θ_(x) ^(i), θ_(y) ^(i)). In FIG. 8, the lightelement 112(f) is shown providing illumination at sample time, t_(i) inthe measurement process. The optical element 130(f) receives and filterslight issuing from specimen 20(f). Light filtered by the optical element130(f) is received at the sensing surface 142 of the radiation detector140(f). The radiation detector 140(f) senses the intensity distributionof the filtered light and captures a low-resolution intensity image ofthe area. Although Fourier ptychographic imaging device 100(f) is shownat a single sample time, t_(i), the Fourier ptychographic imaging device100(f) can operate at a plurality of N sample times, t_(i=1 to N),associated with N incidence angles (θ_(x) ^(i), θ_(y) ^(i)), i=1 to N tocapture N low-resolution two-dimensional intensity images.

High NA Configurations

FIG. 9 depicts an illustration of an orthogonal view of components of aFourier ptychographic imaging device 100(g), according to embodiments.The Fourier ptychographic imaging device 100(g) is an example of an highNA configuration.

In FIG. 9, the high NA Fourier ptychographic imaging device 100(g)comprises a circular variable illuminator 110(g), an optical system130(g) having an objective 134(g) (e.g., microscope objective) and atube lens 132(g), and a radiation detector 140(g). In this illustration,the objective 134(g) is the collection (first) optical element of theoptical system 130(g). The objective 13(g) has a relatively high NA(e.g., in the range of about 0.60 to about 0.75). A sample 20(g) isshown on a specimen surface 126 as provided to the Fourier ptychographicimaging device 100(g).

In FIG. 9, the Fourier ptychographic imaging device 100(g) comprises acircular variable illuminator 110(g) having nine (9) discrete lightelements 112(g) arranged in a single ring. In other cases, the circularvariable illuminator 110(g) may be in the form of a multiple concentricrings, or in other arrangements. In the illustrated example, the angularspacing between adjacent light elements 112(g) is 40 degrees and thediameter of the ring is 40 mm. In other cases, the angular spacingbetween adjacent light elements (e.g., LEDs) may be about 2 degrees. Inother cases, the angular spacing between adjacent light elements (e.g.,LEDs) may be in a range of between about 2 degrees to 40 degrees. Inother cases, the diameter of the ring(s) may be in the range of about 20mm to 40 mm.

In certain aspects, a Fourier ptychographic imaging system may include acircular variable illuminator with light elements arranged in one ormore concentric rings (e.g. 1, 2, 3, etc.). In FIG. 9, for example, thecircular variable illuminator 110(g) comprises light elements in theform of a single ring. The diameters of multi-ring arrangements may bein the range of about 10 mm to about 60 mm. In many cases, the lightelements in each ring are equi-spaced (separated by a uniform angulardifference between adjacent light elements), however, other spacings maybe used. In many cases, each ring will have a different number of lightelements. In other cases, each ring will have the same number of lightelements.

Using a circular variable illuminator with light elements arranged inone or more concentric circles e.g., those with equi-spaced lightelements, can help improve uniformity of overlapping information. Thisuniformity may result in improved image quality as compared with imagesfrom systems that use variable illuminators with light elements in otherarrangements. For example, in cases where the rectangular array variableilluminator has a rectangular grid arrangement of elements, the expandedregion in Fourier space may not be as uniform in the radial direction.As you can see from the illustrations in FIGS. 10A and 10B associatedwith the system using light elements arranged in concentric rings, theexpanded region 280 in Fourier domain is substantially circular so thatthe information in the higher frequencies associated with moving outradially will be substantially uniform. In comparison, an expandedregion associated with a rectangular arrangement of light elements issubstantially rectangular so that the information at the higherfrequencies will not be as uniform.

In FIG. 9, each light element 112(g) is illustrated as an LED, althoughother types of light elements can be used. In this example, each lightelement 112(g) has a radiation source when illuminated. As denoted bythe dotted line, each light element 112(g) sequentially and individuallylights up to provide illumination 114(g) with a wavevector of (kx, ky).In this case, the sample 20(g) can be illuminated from 9 differentincidence angles by illumination provided by the each of the 9 lightelement 112(g). In one example operation, the sample 20(g) isilluminated from 9 different incidence angles at different acquisitiontimes, the optical system 130(g) collects light issuing from theilluminated sample 20(g), the objective lens 134(g) filters lightissuing from the sample based on its acceptance angle, the tube lensfocuses the filtered light to the radiation detector 140(g), and theradiation detector 140(g) captures nine (9) intensity images at theacquisition times.

In FIG. 9, the circular variable illuminator 110(g) is located toprovide illumination 114(g) in a trans-illumination mode i.e.illumination 114(g) is directed through the sample 20(g). In anothercase, the variable illuminator 110(g) may be located to provideillumination in an epi-illumination mode, e.g., located on the same sideof the sample 20(g) as the objective lens 134(g).

In certain aspects, illumination from a variable illuminator at anincidence angle approximates plane wave illumination. Illumination by anoblique plane wave with a wavevector (kx, ky) is generally equivalent toshifting the center of the sample's spectrum by (kx, ky) in the Fourierdomain. Here, k_(x)=k₀·cos x (cosine of angle between illuminationwavevector and x-axis); k_(y)=k₀·cos y (cosine of angle betweenillumination wavevector y-axis); and

$k_{0} = {\frac{2\;\pi}{\lambda}.}$The pupil function (i.e. coherent optical transfer function) of thefiltering optical element (e.g., objective lens 134(g) in FIG. 9) inFourier space can be described as a circular shape low-pass filter witha radius of NA_(obj)·k₀ which is

${NA}*\frac{2\;\pi}{\lambda}$in this case, where NA_(obj) is of the filtering optical element. Thus,each intensity image acquired by the radiation detector based on theapproximated plane wave illumination with wavevector (kx, ky) from thevariable illuminator contains sample's spectrum information centered atabout (kx, ky) in the Fourier domain. With illumination having awavevector of (kx ky) or (k₀·cos x, k₀·cos y), the image captured by thesystem contains spatial frequency information as high ask₀·[NA_(obj)+√{square root over ((cos²x+cos²y))}], where √{square rootover ((cos²x+cos²y))}=NA_(ill) is the numerical aperture of theillumination. The synthesized NA of the system can be described asNA_(syn)=NA_(obj)+NA_(ill).

To exceed unity NA_(sys) in a Fourier ptychographic imaging system,components are configured such that the NA_(obj)+NA_(ill) sums up togreater than 1. For example, by using the high NA configuration shown inFIG. 9 with a circular variable illuminator having a circular ring of 9light elements (e.g., LEDs), the NA_(ill)=0.70 and with an filteringoptical element in the form of an objective lens having NA_(obj)=0.75(e.g., 40×, 0.75NA microscope objective lens), the resulting dryobjective system has a NA_(syn)=1.45 while retaining the field-of-view,and working distance of the objective lens. As another example, by usingthe using the high NA configuration shown in FIG. 9 with an oilimmersion setup having a 100×1.4NA objective for image acquisition andanother 100×1.4NA for illumination (by imaging the light elements at theback focal plane of the objective which could form collimatedillumination with NA_(ill)=1.4), the NA_(sys) could be as high as 2.8.

In some aspects, an iterative recovery process can be used to stitch theinformation at each of these regions associated with the plurality ofincidence angles to expand the information in the Fourier domain tocapture higher frequency information at the outer regions and to captureuniformly overlapping and wider regions of information, which can resultin higher resolution images of the sample. This expansion of theintrinsic NA_(obj) of the filtering optical element may generate anexpanded synthetic NA of the system.

In certain high NA Fourier ptychographic imaging systems describedherein, the filtering optical element has a relatively high NA in orderto capture higher frequency information for each incidence angles, whichcorresponds to a wider circular region for each incidence angle in theFourier domain, which can result in an image having a better resolutionthan about 400 nm. For example, a Fourier ptychographic imaging systemwith the Fourier ptychographic imaging device 110(g) shown in FIG. 9 isa high NA configuration. In this example, the objective lens 134 has arelatively high NA, for example, in a range of about 0.6 to about 0.75.In addition, the variable illuminator 110(g) has nine (9) light elements(e.g., LEDs) in s ring. FIG. 10A is an illustration depicting theexpansion in the Fourier domain for this high NA configuration shown inFIG. 9, of an embodiment. FIG. 10B is the illustration of FIG. 10A shownon a white background for clarification of certain details.

Certain Fourier ptychographic imaging systems described herein useangularly varying illumination to acquire high frequency informationabout the sample. In certain cases, such as with a system having thehigh NA configuration shown in FIG. 9, the system acquires higherfrequency information by using a higher NA filtering optical elementand/or by increasing the range of incidence angles used by the variableilluminator. Using an iterative recovery process (e.g. iterative phaseretrieval process), the high frequency information about the sample canbe “stitched” together in the Fourier domain, such as shown in FIGS. 10Aand 10B, which means that an expanded synthesized NA and finerresolution has been generated in the space domain.

In FIGS. 10A and 10B, the center circular region 250 represents therange of information that can be captured by the objective lens 134(g)(e.g., NA=0.60). Each of the nine (9) overlapping circular regions 260represents the range of information captured by the same objective lens134(g) at oblique angle illumination. Each overlapping circular region260 corresponds to one of the nine (9) different incidence angles. Thecircular region 280 shows the range of information captured by theobjective 134(g) at the (9) different incidence angles. For reference, acircular region 270 is illustrated to show the range of informationcaptured by a unity NA objective. As shown, the circular region 280 ofthe range of information captured by the objective at the (9) differentincidence angles is larger than the circle 270 of the unity NA objectivei.e. the NA is greater than 1.0. That is, by overlapping circularregions in Fourier space, the combined region can form an NA of morethan 1.0. In configurations where the intrinsic NA of the objective maybe lower than 0.6, more LEDs can be arranged (either circularly or in asquare array) to provide enough illumination angle, such that the areainside NA=1.0 can be fully occupied in the Fourier domain.

With oil immersion technology, a conventional microscope can achieve amaximum NA of 1.0. Using a Fourier ptychographic imaging system in ahigh NA configuration, such as with the Fourier ptychographic imagingdevice 100(g) shown in FIG. 9, the NA of the filtering optical elementis relative high and the resulting expanded NA of the system has beenshown to exceed 1.0.

FIG. 11A is an illustration depicting the expansion in the Fourierdomain for an high NA configuration similar to the one shown in FIG. 9,but with a variable illuminator having two concentric circles (rings) oflight elements (four elements on an inner ring and 12 light elements onan outer ring) and with an objective having an NA of 0.50, according toan embodiment. The inner ring has four (4) light elements and the outerring has twelve (12) light elements. FIG. 11B is the illustration ofFIG. 11A shown on a white background for clarification of certaindetails.

In FIGS. 11A and 11B, the center circular region 252 represents therange of information that can be captured by an objective lens havingNA=0.50. The four (4) overlapping circular regions 262 (corresponding tothe inner ring of the variable illuminator) represent the range ofinformation captured by the objective lens with NA=0.50 at oblique angleillumination at four corresponding incidence angles. Each overlappingcircular region 262 corresponds to one of the four (4) differentincidence angles. The twelve (12) overlapping circular regions 264(corresponding to the outer ring of the variable illuminator) representthe range of information captured by the objective lens with NA=0.50 atoblique angle illumination at 12 corresponding incidence angles. Eachoverlapping circular region 264 corresponds to one of twelve (12)different incidence angles.

The circular region 282 shows the expanded range of information capturedby the objective 134 having an NA of 0.50 at 16 different incidenceangles. For reference, a circular region 270 is illustrated to show therange of information captured by a unity NA objective. As shown, thecircular region 282 of the expanded range of information captured by theobjective at the sixteen (16) different incidence angles is larger thanthe circle 270 of the unity NA objective.

FIG. 12 depicts an illustration of an orthogonal view of components of aFourier ptychographic imaging device 100(h), according to an embodiment.The Fourier ptychographic imaging device 100(h) is an example of an highNA configuration. In FIG. 12, the Fourier ptychographic imaging device100(h) comprises a rectangular array variable illuminator 110(h), anoptical system 130(h) having an objective 134(h) (e.g., microscopeobjective) and a tube lens 132(h), and a radiation detector 140(h). Inthis illustration, the objective 134(h) is the collection (first)optical element of the optical system 130. The objective 132(h) has arelatively high NA (e.g., in the range of about 0.50 to about 0.75). Asample 20(h) is shown on a specimen surface 126(h) as provided to theFourier ptychographic imaging device 100(h).

In FIG. 12, the rectangular array variable illuminator 110(h) is locatedto provide illumination 114(h) in a trans-illumination mode i.e.illumination 114(h) is directed through the sample 20(h). In anothercase, the variable illuminator 110(h) may be located to provideillumination in an epi-illumination mode, e.g., located on the same sideof the sample 20(h) as the objective lens 134(h).

In FIG. 12, the Fourier ptychographic imaging device 100(h) comprises avariable illuminator 110(h) having light elements 112(h) in arectangular grid arrangement with 225 equi-spaced light elements thatcorresponds to a 15×15 square array. Other numbers and arrangements oflight elements can be used. In the illustrated example, the spacingbetween adjacent light elements 112(h) is in a range of about 2 degreesto about 40 degrees.

In FIG. 12, each light element 112(h) is illustrated as an LED, althoughother types of light elements can be used. In this example, each lightelement 112(h) has a radiation source when illuminated. Duringoperation, each light element 112(h) sequentially and individuallylights up to provide illumination 114(h) with a wavevector of (kx, ky).In this case, the sample 20(h) can be illuminated from 225 differentincidence angles by illumination provided by the each of the 225 lightelement 112(h). In one example operation, the sample 20(h) isilluminated from 225 different incidence angles at 225 differentacquisition times, the optical system 130(h) collects light issuing fromthe illuminated sample 20(h), the objective lens 134(h) filters lightissuing from the sample based on its acceptance angle, the tube lensfocuses the filtered light to the radiation detector 140(h), and theradiation detector 140(h) captures 225 intensity images at the 225acquisition times.

Epi-Illumination Configurations

FIGS. 13, 14, 15, and 16 depict schematic diagrams of side views ofcomponents of reflection-mode configurations (configurations inepi-illumination mode) of Fourier ptychographic imaging devices,according to embodiments. Each of these Fourier ptychographic imagingdevices are configured to locate a variable illuminator on the sameplane (e.g., FIG. 13) or behind the plane (e.g. FIGS. 14, 15, and 16) ofthe imaging optics. These illustrated devices are shown inepi-illumination mode. Some examples of primary applications forreflection-mode configurations include metal or semiconductor surfaceinspection, including semiconductor wafer, chip, and/or electroniccircuit board inspection, among others. Secondary applications mayextend to systems that can be applied in epi-illumination mode such ashand-held cameras with a modified flash system, or satellite imagery.

FIG. 13 depicts an illustration of an orthogonal view of components of aFourier ptychographic imaging device 100(i), according to certainembodiments. The Fourier ptychographic imaging device 100(i) comprises acircular variable illuminator 110(i), an optical system 130(i)comprising a filtering optical element in the form of an imaging lens137(i), and a radiation detector 140(i) having a detector plane 142. Asample 20(i) is shown on a specimen surface 126(i) as provided to theFourier ptychographic imaging device 100(i).

In FIG. 13, the imaging lens 137(i) has a focal length f, a radius r,and an acceptance angle 2θ_(A). The imaging lens 137(i) may have an NAin the range of about 0.60 to about 0.75. In the illustrated example,the imaging lens 137(i) may be similar to a large camera lens so thatthe working distance d_(o) is large such as, for example, about 10-20cm. In other examples, a smaller lens may be uses, such as a microscopelens, in which case the working distance d_(o) would be smaller such as,for example, 2-3 cm.

In FIG. 13, the circular array variable illuminator 110(i) compriseslight elements 112(i) (e.g., LEDs) arranged in 12 concentric rings(e.g., circular LED rings) equally spaced between each ring and centeredaround a central axis and around the imaging lens 137(i). Other numbersconcentric rings may be used in other cases such as 1, 2, 3, 4, 5, 6,etc. In this illustrated example, the light elements 112(i) are locatedat the sample plane of the imaging lens 137(i). In other cases, thelight elements 112(i) may be at an offset plane, but remain on the sameside of the sample 20(i) as the imaging lens 137(i) in order to provideillumination in a epi-illumination mode. In the illustrated example, therings are equi-spaced from each other with a radial spacing defined asΔr. In this illustrated example, the Fourier ptychographic imagingdevice 100(i) has a variable illuminator 110(i) that is located at adistance, equal to the working distance d_(o), above the sample 20(i) toprovide epi-illumination mode.

In FIG. 13, the resolution Fourier ptychographic imaging device 100(i)is shown at a single illumination time and/or acquisition time. At thistime, a single light element 112(i) of the variable illuminator 110(i)is activated to provide illumination 114(i) at an incidence angle ofθ_(B) with a wavevector of (kx,ky). At other times, the other lightelements 112(i) may be providing illumination. In an example operationof a system comprising the variable illuminator of the Fourierptychographic imaging device 100(i), the variable illuminator 110(i)generates illumination 114(i) to the sample 20(i) at a plurality of Nincidence angles. The imaging lens 137(i) receives light from the sample20(i) within its acceptance angle to filter the light. The opticalsystem 130(i) propagates the filtered light to the radiation detector140(i), which measures an intensity distribution to capture an intensityimage at different incidence angles.

The illustrated example also includes a distance d_(i) between theimaging lens 137(i) and the radiation detector 140(i) and a workingdistance d₀ between the imaging lens 137(i) and the sample 20(i). In oneexample, the Fourier ptychographic imaging device 100(i) may have thefollowing relative dimensions: f=5 cm; d_(i) 7.02 cm; d_(o)=17.3 cm;r=0.25 cm; θ_(B)=30 degrees; and θ_(A)=3 degrees.

The Fourier ptychographic imaging device 100(i) of FIG. 13 includes avariable illuminator 110(i) that does not have light elements at thecenter where the imaging lens 137(i) is located. Without light elementsat the center, the images generated by the device 110(i) with thisilluminator 110(i) will not include low spatial frequencies. In someapplications, such as characterization of slowly-varying phase objects,or when accurate knowledge of the reflectance from the entire objectsurface is required, this low spatial frequency information may bevaluable. The configuration shown in FIG. 13 has a large workingdistance and a simple design with few components. Since theconfiguration does not collect information at low spatial frequencies,this configuration is ideally suited for imaging of high resolutionfeatures or defects, for example, in chip inspection applications.

FIG. 14 depicts an illustration of an orthogonal view of components of aFourier ptychographic imaging device 100(j), according to certainembodiments. Certain components of the Fourier ptychographic imagingdevice 100(j) are similar to those of the Fourier ptychographic imagingdevice 100(i) shown in FIG. 13.

In FIG. 14, the Fourier ptychographic imaging device 100(j) isconfigured to capture low spatial frequencies that may be omitted by theconfiguration shown in FIG. 13. This Fourier ptychographic imagingdevice 100(j) is configured to capture low spatial frequencies bycomprising a beamsplitter 139(j) and a second smaller set of concentricrings 110(j)(2) of light elements 112(j)(2) on the other side of theimaging lens 138(j) (imaging optics) so that the light elements112(j)(2) are directed toward the image plane of the imaging optics. Thesecond set of light elements 112(j)(2) are focused through the imagingoptics to illuminate the sample with a plane wave at the sample plane.In certain cases, the configuration shown in FIG. 14 includes a largeraperture than the configuration shown in FIG. 13. The configurationshown in FIG. 14 may provide a large working distance as well.

In FIG. 14, the Fourier ptychographic imaging device 100(j) comprises avariable illuminator including a first set of concentric rings 110(j)(1)and a second set of concentric rings 110(j)(2), an optical systemincluding comprises an imaging lens 138(j), a beam splitter 139(j), anda radiation detector 140(j) having a detector plane 142(j). A sample20(j) is shown on a specimen surface 126 provided to the Fourierptychographic imaging device 100(j) during an acquisition process. Theillustrated example shows a working distance d_(o) between the imaginglens 138(j) and the sample 20(j). The illustrated example also includesa distance d_(i) between the imaging lens 138(j) and the radiationdetector 140(j).

The beam-splitter 139(j) is configured to transmit half the illuminationincident at a 45 degree angle to the beam-splitter 139(j) and notabsorbed by the beam-splitter 139(j). The remaining half of the incidentillumination (not absorbed) is reflected by the beam-splitter 139(j).For example, the beam splitter 139(j) may be comprised of a sheet ofglass or other substrate with a coating designed to control the lightaccordingly. As another example, a beam splitter may be a half-silveredmirror with a continuous thin coating of reflective material (e.g.,metal). Another example is a swiss cheese beam splitter which has adiscontinuous coating with holes to obtain the desired ratio ofreflection to transmission.

The imaging lens 138(j) has a focal length f, a radius r, and anacceptance angle of 2θ_(A). In the illustrated example, the imaging lens138(j) is configured to filter light by accepting light within itsacceptance angle, 2θ_(A). Examples of values that can be used in theillustrated configuration are: f=6 cm, r=1 cm, and θ_(A)=5 degrees.Other focal lengths, radii, and acceptance angles can be used. Tomaintain a large lens-sample distance, the imaging lens 138(j) has arelatively low NA in the range of about 0.1 to about 0.3. In theillustrated example, the imaging lens 138(j) has an NA of about 0.16,which is a relatively low NA (e.g., about 0.08, about 0.09, about 0.10,in a range of between about 0.07 to about 0.20, etc.).

In the illustrated example, the imaging lens 138(j) may be, for example,a large camera lens having a focal length f of 6 cm and a radius r of 2cm. If using a large camera lens, the Fourier ptychographic imagingdevice 100(j) will have a corresponding large working distance d_(o)such as, for example, about 10-20 cm. In other examples, a smaller lensmay be uses such as a microscope lens, in which case the workingdistance d_(o) would be smaller such as, for example, 2-3 cm. In theillustrated example, d_(o)=12 cm and d_(i)=12 cm; other values may beused.

In FIG. 14, the optical path distance between the beam splitter 139(j)and the second set of concentric rings 110(j)(2) is designated as b andthe optical path distance between the beam splitter 139(j) and theimaging lens 138(j) is designated as a. In the illustrated example, theoptical system is configured so that the imaging lens 138 is located ata combined optical path distance of a+b=f from the second set ofconcentric rings 110(j)(2).

In FIG. 14, the variable illuminator of the Fourier ptychographicimaging device 100(j) comprises two sets of concentric rings (e.g.,circular LED arrays) of light elements: a first set of twelve (12)equally-spaced concentric rings 110(j)(1) (e.g., a first LED array) anda second set of eight (8) equally-spaced concentric rings 110(j)(2)(e.g., a second LED array). Other numbers of concentric rings may beused in other cases such as 1, 2, 3, 4, 5, 6, etc. The first set ofconcentric rings 110(j)(1) comprises light elements 112(f) located atthe plane of the imaging lens 138(j) and centered around the imaginglens 138(j). In other cases, the light elements 112(f) may be at one ormore offset planes on the same side of the sample 20(j) as the imaginglens 138(j) to be configured for illumination in a epi-illuminationmode. The first set of concentric rings 110(j)(1) are equally-spacedwith a uniform radial spacing of Δr₁. The second set of concentric rings110(j)(1) are equally-spaced with a uniform radial spacing of Δr₂. Thefirst set of concentric rings 110(j)(1) are located at a distance, equalto the working distance d_(o), above the sample 20(j).

In this illustrated example, the first set of concentric rings 110(j)(1)are centered around a central axis of the imaging lens 138(j) so thatthe first set does not have light elements 112(j)(1) across the centerof the imaging lens 138(j). The second set of first set of concentricrings 110(j)(1) has light elements 112(j)(2) configured to provideillumination reflected by the beam splitter 139(j) through the imaginglens 138(j). The second set of concentric rings 110(j)(2) compriseslight elements 112(j)(2) located at a plane that is at a combinedoptical path (a+b) of a focal length f from the imaging lens 138(j).

In FIG. 14, the Fourier ptychographic imaging device 100(j) is shown ata single illumination time and/or acquisition time. At this time, asingle light element 112(j)(1) from the from the first set of concentricrings 110(j)(1) is shown providing illumination 114(j) at an incidenceangle of θ_(B) with a wavevector of (kx,ky). At other times, the otherlight elements 112(j)(1) or 112(j)(2) may be providing illumination. Ifone of the light elements 112(j)(2) is illuminated, incident light isreceived by the beam splitter 139(j). Half the incident light receivedat the beam splitter 139(j) (and not absorbed) is reflected to theimaging lens 138(j) which propagates illumination to the sample 20(j).Since the beam splitter 139(j) passes half the incident illumination, incertain aspects, each of the light elements 112(j)(2) of the second setof concentric rings 110(j)(2) has a light source with about two (2)times (2×) the intensity of the light source of each of the lightelements 112(j)(1) of the first set of concentric rings 110(j)(1). Incertain cases, the intensity from the light elements 112(j)(2) may beadjusted to provide incident illumination at the sample 20 of about thesame intensity as the incident illumination provided by the lightelements 112(j)(1).

In an example operation of a system comprising the variable illuminatorof the Fourier ptychographic imaging device 100(j), the light elements112(j)(1) and 112(j)(2) of the variable illuminator generateillumination directed to the sample at a plurality of N incidenceangles. Light reflected by the sample 20(j) is received at the imaginglens 138(j). The imaging lens 138(j) receives light within itsacceptance angle to filter the light. The imaging lens 138(j) propagatesincident light to the beam splitter 138(j). Half the incident light fromthe imaging lens 138(j) is transmitted through the beam splitter 138(j)and propagated to the radiation detector 140(j), which measures theintensity distribution at different acquisition times to captures aplurality of intensity images at different incidence angles.

FIG. 15 and FIG. 16 depict illustrations of orthogonal views ofcomponents of a Fourier ptychographic imaging device 100(k), accordingto certain embodiments. FIG. 15 illustrates the illumination scheme andFIG. 16 illustrates the collection scheme of the Fourier ptychographicimaging device 100(k).

In FIG. 15 and FIG. 16, the Fourier ptychographic imaging device 100(k)comprises a variable illuminator 110(k) comprises twelve (12) concentricrings 110(k)(1) of light elements 112(k), an optical system, and aradiation detector 140(k) having a detector plane 142(k). The variableilluminator 110(k) comprises twelve (12) concentric rings 110(e)(1) oflight elements 112(k). Other numbers of concentric rings may be usedsuch as, for example, 1, 2, 3, 4, 5, 6, 7, 8, 9, . . . 13, 14, 15, etc.The outermost concentric ring has a width w. The optical systemcomprises a objective 134(k) (e.g., microscope objective) with a focallength f, a tube lens 132(k), a secondary lens 138(k), and a beamsplitter 139(k). Although the objective 134(k) is illustrated here as amicroscope objective, another objective may be used. A sample 20(k) isshown on a specimen surface 126 as provided to the Fourier ptychographicimaging device 100(k). The illustrated example shows a working distanced_(o) between the objective 134(k) and the sample 20(k). In theillustrated example, a microscope objective may be used so that theconfiguration has a short working distance such as, for example, 2-3 cm.One operational range could be with a 0.08NA 2× objective lens with a ˜2cm working distance. Another could be with a 20×0.5 NA objective lenswith a ˜2 mm working distance.

In the illustrated configuration, the entire variable illuminator 110(k)(e.g., LED array) is located behind the objective 134(k) (primaryimaging optics) and a secondary lens 130(k) is used to image thevariable illuminator 110(k) to a back focal plane of the objective. InFIG. 15 and FIG. 16, the optical path distance between the beam splitter139(k) and the secondary lens 138(k) is d_(i1), the optical pathdistance between the secondary lens 138(k) is d₂, and the optical pathdistance between the beam splitter 139(k) and the back focal plane ofthe objective 134(k) is d_(i2). In FIG. 15, an image 136 of the variableilluminator 110(k) is shown at the back focal plane 135 an opticaldistance of a focal length F from the back of the objective 134(k). Toassure that the variable illuminator 110(k) image is formed on the backfocal plane 135(k) of the objective 134(k), the components of theoptical system are located so that the optical path distances followthis equation: 1/f=1/d₂+1/(d_(i1)+d_(i2)). FIG. 16 also shows an opticaldistance of d_(s) from the tube lens 132(k) to the radiation detector140(k) and an d_(t) between from the tube lens 132(k) to the back of theobjective 134(k). The illustrated example includes an objective 134(k)that is a 2× microscope objective. In other examples, other objectivesmay be used. An example of values that can be used in the illustratedconfiguration are w=10 cm, the d₂=20 cm, the d_(i1)+d₁₂=2 cm, and thef=1.9 cm. Other values can be used.

The beam-splitter 139(k) is configured to transmit half the illuminationincident at a 45 degree angle to the beam-splitter 139(k) and notabsorbed by the beam-splitter 139(k). The remaining half of the incidentillumination (not absorbed) is reflected by the beam-splitter 139(k).For example, the beam splitter 139(k) may be comprised of a sheet ofglass or other substrate with a coating designed to control the lightaccordingly. As another example, a beam splitter may be a half-silveredmirror with a continuous thin coating of reflective material (e.g.,metal). Another example is a swiss cheese beam splitter which has adiscontinuous coating with holes to obtain the desired ratio ofreflection to transmission.

In FIG. 15, the resolution Fourier ptychographic imaging device 100(k)is shown at a single illumination time. At this time, a single lightelement 112(k) of the variable illuminator 110(k) is activated toprovide illumination at an incidence angle of θ_(B) with a wavevector of(k_(x)k_(y)). At other times, other light elements 112(k) may beproviding illumination at other incidence angles. Each light element112(k) includes a light source that can provide illumination (e.g.,approximately plane wave illumination) at a particular incidence angleto the sample 20(k).

As shown in FIG. 15, during an operation of a system comprising thevariable illuminator of the Fourier ptychographic imaging device 100(k),different light elements 112(k) of the variable illuminator 110(k) areilluminated at different times. The secondary lens 138(k) receivesillumination from the illuminated light element(s) 112(k) and propagatesthe illumination to the beam splitter 139(k). The beam splitter 139(k)transmits half the incident light and reflects half the incident light.The objective 134(k) propagates incident light to the sample toilluminate it at a plurality of N incidence angles at different times.As shown in FIG. 16, during the operation of the system, light issuingfrom the sample 20(k) is received by the objective 134(k) acting as thefiltering optical element of the optical system. The objective 134(k)propagates light to the beam splitter 139(k), which transmits half thelight, not absorbed, and reflects the remainder. The tube lens 132(k)receives light passing through the beam splitter 139(k) and propagateslight to the radiation detector 140(k). The radiation detector 140(k)measures the intensity distribution at different acquisition times tocapture a plurality of intensity images at different incidence angles.

III. Fourier Ptychographic Imaging Methods

In certain aspects, a Fourier ptychographic imaging method comprises ameasurement/acquisition process, a recovery/reconstruction process, andan optional display process. During the measurement process, the sampleis illuminated from a plurality of N incidence angles (θx_(i,j),θy_(i,j)), i=1 to n, j=1 to m, (N=n×m) using a variable illuminator.During this process, the optical system filters the light issuing fromthe illuminated sample to propagate filtered light to the radiationdetector and the radiation detector receives the filtered light andacquires a plurality of M intensity images, I_(k,l), k=1 to o and j=1 top, where M=o×p. In certain cases, an intensity image is captured at eachincidence angle. In certain aspects, the variable illuminator may bedesigned to generate illumination at certain incidence angles thatgenerate intensity data that corresponds to regions that overlap in theFourier domain by a certain amount and also cover outer higher frequencyarea. During the recovery process, the M intensity images areiteratively combined in the Fourier domain to generate higher-resolutionimage data (intensity and/or phase). At each iteration, a filter isapplied in the Fourier domain for a particular plane wave incidenceangle, an inverse Fourier transform is applied to generate a lowerresolution image, the intensity of the lower resolution image isreplaced with an intensity measurement from the radiation detector, aFourier transform is applied, and the corresponding region in Fourierspace is updated. During the optional display process, an image (e.g.,higher-resolution image, acquired intensity image, etc.) and/or otheroutput may be provided on a display. Generally, these methods alternatebetween two working domains: the spatial (x-y) domain and the Fourier(kx-ky) domain, where k represents the wavenumber.

In certain aspects, Fourier ptychographic imaging methods may comprise aphase retrieval technique that uses angular diversity to recover complexsample images. The recovery process alternates enforcement of knownimage data acquired in the spatial domain and a fixed constraint in theFourier domain. This phase retrieval recovery can be implemented usingvarious methods such as, for example, an alternating projectionsprocedure, a convex reformulation of the problem, or any non-convexvariant in-between. Instead of needing to translate a sample laterally(i.e. applying translational diversity), Fourier ptychographic imagingsystems use methods that vary the spectrum constraint in the Fourierdomain to expand the Fourier passband beyond that of a single capturedimage to recover a higher-resolution sample image.

In some cases, Fourier ptychographic imaging methods may also comprisean optional aberration correction process. An example of an aberrationcorrection process is a re-focusing (propagating) process. Such arefocusing process may be useful where the sample was placed at a sampleplane at z=z₀ where the in-focus plane of the optical element is locatedat position z=0. In other words, the image captured of the sample is notthe image at the sample plane, but is the sample profile propagated by adistance of −z₀ from the in-focus plane of the optical element. In thesecases, the method may re-focus the sample by propagating the image databy the z₀ distance back to the sample plane, without having tomechanically move the sample in the z-direction. The re-focusing(propagating) step(s) can be performed by multiplying a phase factor inFourier space.

With reference to certain illustrated examples, subscript “h” refers tohigher-resolution, subscript “l” refers to lower resolution intensity,subscript “f” refers to focused position, subscript “m” refers tomeasured, and subscript “s” refers to sampled.

FIG. 17 is a flowchart depicting steps of a Fourier ptychographicimaging method, according to certain embodiments. This method isperformed by a Fourier ptychographic imaging system such as, forexample, the system 10 described with reference to FIG. 1. The Fourierptychographic imaging method comprises a measurement process (steps1100, 1200, and 1300), a recovery process (steps 1400 and 1500), and anoptional display process (step 1600).

At step 1100, a variable illuminator provides illumination to a samplefrom a plurality of N incidence angles (θx_(i,j), θy_(i,j)), i=1 to n,j=1 to m, at N sample times. In some cases, the variable illuminatorcontrols the illumination provided to the sample based on illuminationinstructions. The illumination instructions may define the order of theillumination angles and the associated illumination time. The wavevector in x and y directions can be denoted as wavevector kx_(i,j),ky_(i,j).

In certain aspects, the variable illuminator may provide illumination ofdifferent wavelengths at different sample times. For example, thevariable illuminator may provide RGB illumination of three wavelengthsλ₁, λ₂, and λ₃ corresponding to red, green, blue colors, respectively,at different sample times, for example, in a color imaging embodiment.

In some cases, the variable illuminator is configured to provide planewave illumination. Plane wave illumination with a wavevector, k_(x),k_(y), in the spatial domain, is equivalent to shifting the center ofthe image spectrum by (k_(x), k_(y)) in the Fourier domain. In thisrespect, the intensity image data in the Fourier domain is shifted fromnormal incidence image data by (kx, ky), which corresponds to theincidence angle (θ_(x), θ_(y)) applied by the variable illuminator.

At step 1200, the optical system collects light issuing from the sampleand propagates it to the radiation detector. The optical systemcomprises a filtering optical element(s) that filters the light. Forexample, a filtering optical element may be an objective lens collectinglight issuing from an illuminated sample. In this case, the objectivelens filters the light issuing from the sample by only accepting lightincident at a range of angles within its numerical aperture (NA). InFourier space, the filtering function of a filtering optical elementsuch as an objective lens may be represented by a circular pupil withradius of NA×k₀, where k₀=2π/λ is the wave number in vacuum. That is,the Fourier ptychographic imaging method may update in Fourier spacecircular regions defined by this filtering function and the differentincidence angles. In certain cases, the filtering optical element andits associated filtering function omits data outside the circular pupilregion.

At step 1300, the radiation detector receives light propagated by theoptical system and captures a snapshot intensity distributionmeasurement at each of the M sample times, t_(k), k=1 to M, to acquire aplurality of M intensity images, I_(k,l), k=1 to o and j=1 to p,associated with different incidence angles. Each intensity image sampledby the radiation detector is associated with a region in Fourier space.In many aspects, the variable illuminator is configured to provideillumination from incidence angles that will generate overlapping areasbetween neighboring (adjacent) regions (e.g., circular pupil regions) inFourier space. In one aspect, the variable illuminator is designed toprovide an overlapping area between neighboring regions of 2% to 99.5%of the area of one of the regions. In another aspect, the variableilluminator is designed to provide an overlapping area betweenneighboring regions of 65% to 75% of the area of one of the regions. Inone aspect, the variable illuminator is designed to provide anoverlapping area between neighboring regions of about 65% of the area ofone of the regions.

At steps 1400 and 1500, a higher-resolution image of the sample may berecovered by updating the sample spectrum with overlapping datasetsbased on the M intensity distribution measurements acquired at step1300. The M intensity images, I_(k,l) k=1 to o and j=1 to p correspondto different incidence angles indexed by illumination wavevectorky_(i,j), ky_(i,) i=1 to n, and j=1 to m.

At step 1400, an initial sample spectrum S(u) or and/or pupil functionP(u) are initialized in the Fourier domain. For example, ahigher-resolution image may be initialized in the spatial domain, and aFourier transform then applied to the initial value to obtain alsoinitial sample spectrum in the Fourier domain which is also referred toas an initial Fourier transformed image Ĩ_(h). The initial samplespectrum may be an initial guess. In some cases, the initial guess maybe determined as a random complex matrix (for both intensity and phase).In other cases, the initial guess may be determined as an interpolationof the low-resolution intensity measurement with a random phase. Anexample of an initial guess is φ=0 and I_(h) interpolated from anylower-resolution image of the sample area. Another example of an initialguess is a constant value. The Fourier transform of the initial guesscan be a broad spectrum in the Fourier domain.

At step 1500, a sample spectrum in Fourier space is constructed byiteratively updating regions in Fourier space with lower-resolutiondatasets based on the intensity measurements captured at differentillumination incidence angles and then inverse Fourier transforming thesolution to a higher resolution image of the sample in the spatialdomain. In many cases, at least portions of step 1500 may be implementedusing a processor (e.g., processor 210 of the system 10).

At optional step 1600, the display may receive image data such as thehigher-resolution image data and/or other data from the processor, anddisplay the data on a display (e.g., display 230 in FIG. 1).

A. Aberration Correction Using Pre-Calibrated Aberration

In certain aspects, the recovery process step 1500 may comprise anaberration correction process that introduces a phase map to thefiltering function to compensate for aberrations at the pupil planeduring the iterative image recovery process. FIG. 18 is a flowchartdepicting an example of sub-steps of step 1500 of the Fourierptychographic imaging method of FIG. 17 that optionally comprises anaberration correction process, according to certain aspects. In oneexample, the method described with respect to the flowchart in FIG. 18can use an initial pupil function that is estimated from apre-calibrated aberration.

In the illustrated flowchart, the optional aberration correction processcomprises incorporating compensation at the two multiplication steps1610 and 1645. Step 1610 models the connection between the actual sampleprofile and the captured intensity data (with includes aberrations)through multiplication with a pupil function: e^(i·φ(k) ^(x) ^(,k) ^(y)⁾. Step 1645 inverts such a connection to achieve an aberration-freereconstructed image. For example, aberration correction can correctsample defocus. In certain cases, sample defocus may be essentiallyequivalent to introducing the following defocus phase factor to thepupil plane (i.e., a defocus aberration):e ^(i·φ(k) ^(x) ^(,k) ^(y) ⁾ =e ^(i)√{square root over (^((2π/λ)) ²^(−k) ^(x) ² ^(k) ^(y) ² )}^(·z) ⁰ ,k _(x) ² +k _(y) ²<(NA·2π/λ)²  (Eqn.1)where kx and ky are the wavenumbers at the pupil plane, z₀ is thedefocus distance, and NA is the numerical aperture of the filteringoptical element.

At step 1605, a processor performs filtering of the higher-resolutionimage √{square root over (I_(h))}e^(iφ) ^(h) in the Fourier domain togenerate a lower-resolution image √{square root over (I_(h))}e^(iφ) ^(l)for a particular plane wave incidence angle (θ_(x) ^(i), θ_(y) ^(i))with a wave vector (kx_(i,j), ky_(i,j)). The Fourier transform of thehigher-resolution image is Ĩ_(h) and the Fourier transform of thelower-resolution image for a particular plane wave incidence angle isĨ_(l). In the Fourier domain, the method filters a region from thespectrum Ĩ_(h) of the higher-resolution image √{square root over(I_(h))}e^(iφ) ^(h) . In cases with a filtering optical element in theform of an objective lens, this region is a circular pupil aperture witha radius of NA*k₀, where k₀ equals 2π/λ (the wave number in vacuum),given by the coherent transfer function of an objective lens. In Fourierspace, the location of the region (e.g., location of center of circularregion) corresponds to the corresponding incidence angle. For an obliqueplane wave incidence with a wave vector (kx_(i,j), ky_(i,j)), the regionis centered about a position (kx_(i,j), ky_(i,j)) in the Fourier domainof √{square root over (I_(h))}e^(iφ) ^(h) .

At optional step 1610, the processor may multiply by a phase factore^(i·φ(k) ^(x) ^(,k) ^(y) ⁾ in the Fourier domain as part of aberrationcompensation.

At step 1625, an inverse Fourier transform is taken to generate thelower resolution image region √{square root over (I_(lf))}e^(iφ) ^(l) .

At step 1630, the computed amplitude component √{square root over(I_(lf))} of the lower-resolution image region at the in-focus plane,√{square root over (I_(lf))}e^(iφ) ^(lf) , is replaced with thelow-resolution intensity measurement √{square root over (I_(lfm))}captured by the radiation detector. This forms an updated lowerresolution image: √{square root over (I_(lfm))}e^(iφ) ^(lf) . A Fouriertransform is then applied to the updated lower resolution image data.

At optional step 1645, an inverse phase factor e^(−i·φ(k) ^(x) ^(,k)^(y) ⁾ is applied in the Fourier domain.

At step 1650, the corresponding region of the higher-resolution solution√{square root over (I_(h))}e^(iφ) ^(h) in the Fourier domaincorresponding to incidence wave vector (k_(x), k_(y)) is updated withthe updated lower resolution image data.

At step 1660, it is determined whether steps 1605 through 1650 have beencompleted for the different incidence angles associated with thecaptured images. If steps 1605 through 1650 have not been completed forthese different incidence angles, steps 1605 through 1650 are repeatedfor the next incidence angle. The next incident angle is typically thenext adjacent angle. In certain aspects, the neighboring (adjacent)regions are overlapping in Fourier space and are iteratively updated(e.g., by repeating steps 1605 through 1650 for each adjacent incidenceangle). At the overlapping area between adjacent regions, there is databased on multiple samplings over the same Fourier space. The incidenceangles of the illumination from the variable illuminator determine theoverlapping area between the regions. In one example, the overlappingarea between neighboring regions is in the range of about 2% to 99.5% ofthe area of one of the corresponding neighboring regions. In anotherexample, the overlapping area between neighboring regions is in therange of about 65% to 75% of the area of one of the correspondingneighboring regions. In another example, the overlapping area betweenneighboring regions is about 65% of the area of one of the correspondingneighboring regions. In another example, the overlapping area betweenneighboring regions is about 70% of the area of one of the correspondingneighboring regions. In another example, the overlapping area betweenneighboring regions is about 75% of the area of one of the correspondingneighboring regions. In certain embodiments, each overlapping region hasthe same area.

At step 1670, it is determined whether a higher-resolution image datahas converged. For example, a processor may determine whether thehigher-resolution image data may have converged to be self-consistent.In one case, a processor compares the previous higher-resolution imagedata of the previous iteration or initial guess to the presenthigher-resolution data, and if the difference is less than a certainvalue, the image data may have converged to be self-consistent. If it isdetermined that the image data has not converged, then steps 1605through 1670 are repeated. In one case, steps 1605 through 1670 arerepeated once. In other cases, steps 1605 through 1670 are repeatedtwice or more.

If the image data has converged, the converged image data in Fourierspace is transformed using an inverse Fourier transform to the spatialdomain to recover a higher-resolution image √{square root over(I_(h))}e^(iφ) ^(h) . If it is determines that the solution hasconverged at step 1670, then the method may proceed to optional step1600 or the method may end.

FIG. 19 is a flowchart depicting an example of sub-steps of step 1500shown in FIG. 17, according to an embodiment. These sub-steps comprisean optional aberration correction process that corrects for defocus. InFIG. 19, step 1500 comprises step 1510, step 1530, step 1550, step 1560,step 1570, step 1580, and step 1590. In aspects that include aberrationcorrection, step 1500 may further incorporate compensation at the twomultiplication optional steps 1520 and 1540. For example, optional steps1520 and 1540 can be used to focus an out-of focus sample that isout-of-focus by the amount of z₀.

At step 1510, a processor performs filtering of the higher-resolutionimage √{square root over (I_(h))}e^(iφ) ^(h) in the Fourier domain togenerate a lower-resolution image √{square root over (I_(l))}e^(iφ) ^(l)for a particular plane wave incidence angle (θx_(i,j), θy_(i,j)) with awave vector (kx_(i,j), ky_(i,j)). The Fourier transform of thehigher-resolution image is Ĩ_(h) and the Fourier transform of thelower-resolution image for a particular plane wave incidence angle isĨ_(l). In the Fourier domain, the method filters a region from thespectrum Ĩ_(h) of the higher-resolution image √{square root over(I_(h))}e^(iφ) ^(h) . In cases with a filtering optical element in theform of an objective lens, this region is a circular pupil aperture witha radius of NA*k₀, where k₀ equals 2λ/λ (the wave number in vacuum),given by the coherent transfer function of an objective lens. In Fourierspace, the location of the region (e.g., location of center of circularregion) corresponds to the corresponding incidence angle. For an obliqueplane wave incidence with a wave vector (kx_(i,j), ky_(i,j)), the regionis centered about a position (kx_(i,j), ky_(i,j)) in the Fourier domainof √{square root over (I_(h))}e^(iφ) ^(h) .

At optional step 1520, the low-resolution image, √{square root over(I_(l))}e^(iφ) ^(l) is propagated in the Fourier domain to an in-focusplane at z=0 of the optical system to determine the lower-resolutionimage at the focused position: √{square root over (I_(lf))}e^(iφ) ^(lf). In one case, optional step 1520 can be performed by Fouriertransforming the low-resolution image √{square root over (I_(l))}e^(iφ)^(l) , multiplying by a phase factor in the Fourier domain, and inverseFourier transforming to obtain √{square root over (I_(lf))}e^(iφ) ^(lf). In another case, optional step 1520 can be performed by themathematically equivalent operation of convolving the low-resolutionimage √{square root over (I_(l))}e^(iφ) ^(l) with thepoint-spread-function for the defocus. In another case, optional step1520 can be performed as an optional sub-step of step 1510 bymultiplying Ĩ_(l) by a phase factor in the Fourier domain beforeperforming the inverse Fourier transform to produce √{square root over(I_(lf))}e^(iφ) ^(lf) . In certain instances, optional step 1520 neednot be included if the sample is located at the in-focus plane (z=0) ofthe filtering optical element.

At step 1530, the computed amplitude component √{square root over(I_(lf))} of the lower-resolution image at the in-focus plane, √{squareroot over (I_(lf))}e^(iφ) ^(lf) , is replaced with the square root ofthe low-resolution intensity measurement √{square root over (I_(lfm))}measured by the radiation detector. This forms an updated low resolutiontarget: √{square root over (I_(lfm))}e^(iφ) ^(lf) .

At optional step 1540, the updated low-resolution image √{square rootover (I_(lfm))}e^(iφ) ^(lf) may be back-propagated to the sample plane(z=z₀) to determine √{square root over (I_(ls))}e^(iφ) ^(ls) . Incertain instances, optional step 1540 need not be included if the sampleis located at the in-focus plane of the filtering optical element, thatis, where z₀=0. In one case, step 1540 can be performed by taking theFourier transform of the updated low-resolution image √{square root over(I_(lfm))}e^(iφ) ^(lf) and multiplying in the Fourier space by a phasefactor, and then inverse Fourier transforming it. In another case, step1540 can be performed by convolving the updated low-resolution image√{square root over (I_(lfm))}e^(iφ) ^(lf) with the point-spread-functionof the defocus. In another case, step 1540 can be performed as asub-step of step 1550 by multiplying by a phase factor after performingthe Fourier transform onto the updated target image.

At step 1550, a Fourier transform is applied to the updated target imagepropagated to the sample plane: √{square root over (I_(ls))}e^(iφ) ^(ls), and this data is updated in the corresponding region ofhigher-resolution solution √{square root over (I_(h))}e^(iφ) ^(h) in theFourier space corresponding to the corresponding to the incidence wavevector kx_(i,j), ky_(i,j).

At step 1560, it is determined whether steps 1510 through 1560 have beencompleted for the different incidence angles associated with thecaptured images. If steps 1605 through 1650 have not been completed forthese different incidence angles, steps 1510 through 1560 are repeatedfor the next incidence angle. The next incident angle is typically thenext adjacent angle. In certain aspects, the neighboring (adjacent)regions are overlapping in Fourier space and are iteratively updated(e.g., by repeating steps 1510 through 1560 for each adjacent incidenceangle). At the overlapping area between adjacent regions, there is databased on multiple samplings over the same Fourier space. The incidenceangles of the illumination from the variable illuminator determine theoverlapping area between the regions. In one example, the overlappingarea between neighboring regions is in the range of about 2% to 99.5% ofthe area of one of the corresponding neighboring regions. In anotherexample, the overlapping area between neighboring regions is in therange of about 65% to 75% of the area of one of the correspondingneighboring regions. In another example, the overlapping area betweenneighboring regions is about 65% of the area of one of the correspondingneighboring regions. In another example, the overlapping area betweenneighboring regions is about 70% of the area of one of the correspondingneighboring regions. In another example, the overlapping area betweenneighboring regions is about 75% of the area of one of the correspondingneighboring regions. In certain embodiments, each overlapping region hasthe same area.

At step 1570, it is determined whether a higher-resolution image datahas converged. For example, a processor may determine whether thehigher-resolution image data may have converged to be self-consistent.In one case, a processor compares the previous higher-resolution imagedata of the previous iteration or initial guess to the presenthigher-resolution data, and if the difference is less than a certainvalue, the image data may have converged to be self-consistent. If it isdetermined that the image data has not converged, then steps 1510through 1560 are repeated. In one case, steps 1510 through 1560 arerepeated once. In other cases, steps 1510 through 1560 are repeatedtwice or more. If the image data has converged, the converged image datain Fourier space is transformed using an inverse Fourier transform tothe spatial domain to recover a higher-resolution image √{square rootover (I_(h))}e^(iφ) ^(h) . If it is determines that the solution hasconverged at step 1570, then the method may proceed to optional step1600 or the method may end.

In certain aspects, the Fourier ptychographic imaging method describedwith reference to FIG. 17 can include an optional aberration correctionprocess described with reference to either FIG. 18 or FIG. 22. In oneaspect, the Fourier ptychographic imaging method includes the optionalaberration correction process for refocusing described in optional steps1520 and 1540 of FIG. 1 to refocus. The refocusing feature of optionalsteps 1520 and 1540 propagates the image from the in-focus plane z=0 tothe sample plane at z=z₀. Refocusing may be needed when the sample islocated at the sample plane at z=z₀, while the in-focus plane of thefiltering optical element (e.g., objective lens) is located at positionz=0. In other words, refocusing may be needed when the sample isout-of-focus by the amount of z₀.

FIGS. 20A and 20B are schematic illustrations depicting components of aFourier ptychographic imaging device 100(m) in trans-illumination mode,according to an embodiment. The Fourier ptychographic imaging device100(m) comprises a variable illuminator 110(m) is in the form of atwo-dimensional matrix of light elements (e.g. an LED matrix). In FIGS.20A and 20B, a single light element 112(m) of the variable illuminator110(m) at X_(i,j) (x′,y′) is shown as illuminated at the sample timebeing illustrated. The Fourier ptychographic imaging device 100(m)further comprises an optical system 130(m).

In FIG. 20A, the sample 20(m) is depicted as out-of-focus by an amountof −z₀, and optional steps 1520 and 1540 (depicted here as arrows) canbe used to digitally refocus the sample 20(g) to the in-focus plane122(m) as depicted by the dotted line to the in-focus plane 122(m). InFIG. 20B, the sample 20(m) is located at in-focus plane 122(m). In thiscase, optional steps 1520 and 1540 may not be needed.

FIG. 21 is an illustration of steps of the Fourier ptychographic imagingmethod described with reference to FIGS. 17 and 19, according to anembodiment. The left-hand-side image includes two circular regions 22(a)and 22(b) in Fourier space used to generate the higher-resolution imageregion. The circular regions 22(a) and 22(b) may be defined NA of thefiltering optical element based on approximating as circular pupilfunction with a radius of NA*k₀, where k₀ equals 2π/λ (the wave numberin vacuum). For example, each circular region 22(a) and 22(b) may bedefined by the optical transfer function of a 2× objective lens 0.08NA.In FIG. 21, region 22(a) is of a circular low-pass filter shapeassociated with a plane wave incidence angle: θ_(x)=0; θ_(y)=0, i=1 andRegion 22(b) is of a circular low-pass filter shape associated with aplane wave incidence angle: θ_(x)=−21°; θ_(y)=22°. To perform filteringat each incidence angle, data outside the circular region in the Fourierdomain is omitted, which results in a low-resolution data. Thelow-resolution image resulting from filtering based on plane waveincidence angle of θ_(x)=−21; θ_(y)=22° is shown at the topright-hand-side of FIG. 21. The low-resolution image resulting fromfiltering based on plane wave incidence angle of θ_(x)=−21°; θ_(y)=22°is shown at the bottom right-hand-side of FIG. 21. The wave vectors ofthe incidence angles in the x-direction and y-direction are denoted askx and ky respectively.

When implementing the updating step 1550 of FIG. 19 or the updating step1650 of FIG. 17, the method updates the data within the region 22(a) ofthe higher-resolution reconstruction 22(c) corresponding to the normalincidence θ_(x)=0, θ_(y)=0. The method also updates the data within theregion 22(b) of the higher-resolution reconstruction corresponding tothe n^(th) incidence angle θ_(x)=−21°; θ_(y)=22°. The regions areupdated with low-resolution image measurement data.

B. Tile Imaging

In certain aspects, a Fourier ptychographic imaging method may comprisetile imaging to divide the captured intensity images into a plurality oftile images, independently acquire a higher-resolution image for each ofthe tiles, and then combine the higher-resolution tile images togenerate a full field-of-view higher-resolution image. In some cases,the higher-resolution tile images may be combined with an image blendingprocess. An example of an image blending process is alpha blending whichcan be found in PCT publication WO1999053469, entitled “A system andmethod for performing blending using an over sampled buffer,” filed onApr. 7, 1999, which is hereby incorporated by reference in its entirety.Since higher-resolution images of the tiles may be acquiredindependently, this method may be well suited for parallel computing,which may reduce computational time, and may also reduce memoryrequirements. Moreover, the light from each light element may beaccurately treated as a plane wave for each tile. The incidentwavevector for each tile can be expressed as:

$\begin{matrix}{\left( {k_{x}^{i},k_{y}^{i}} \right) = {\frac{2\;\pi}{\lambda}\begin{pmatrix}{\frac{\left( {x_{c} - x_{i}} \right)}{\sqrt{\left( {x_{c} - x_{i}} \right)^{2} + \left( {y_{c} - y_{i}} \right)^{2} + h^{2}}},} \\\frac{\left( {y_{c} - y_{i}} \right)}{\sqrt{\left( {x_{c} - x_{i}} \right)^{2} + \left( {y_{c} - y_{i}} \right)^{2} + h^{2}}}\end{pmatrix}}} & \left( {{Eqn}.\mspace{14mu} 2} \right)\end{matrix}$where (x_(c),y_(c)) is the central position of each tile of the fullfield-of-view low-resolution image, (x_(i),y_(i)) is the position of thei^(th) light element, and h is the distance between the variableilluminator and the sample. Furthermore, this method can assign aspecific aberration-correcting pupil function to each tile in somecases.

FIG. 22 is a flowchart depicting a Fourier ptychographic imaging methodwhich includes tile imaging, according to an embodiment. This method canbe performed by Fourier ptychographic imaging system of certainembodiments. To take advantage of parallel processing capabilities, theprocessor of the system should be configured with parallel processingcapabilities such as, for example, the GPU unit or a processor havingmultiple cores (i.e. independent central processing units).

In FIG. 22 the Fourier ptychographic imaging method comprises ameasurement process (steps 2100, 2200, and 2300), a recovery process(steps 2350, 2400(i-M), 2500(i-M), 2590), and an optional displayprocess (step 2600). The measurements process (steps 2100, 2200, and2300) and optional display process (step 2600) are described withreference to FIG. 17.

At step 2350, the processor divides the full field-of-view into aplurality of tiles such as, for example, a two-dimensional matrix oftiles. The dimensions of a two-dimensional square matrix of tiles may bein powers of two such as, for example, a 256 by 256 matrix, a 64×64matrix, etc. In one example, the processor may divide up a full field ofview of 5,280×4,380 pixels into tiles having an area of 150×150 pixels.

Next, the processor initializes the higher-resolution image: √{squareroot over (I_(h))}e^(iφ) ^(h) in the spatial domain for each tile (1 toT) independently using parallel computing (step 2400(1) . . . step2400(T)). A Fourier transform is applied to the initial guess. In somecases, the initial guess may be determined as a random complex matrix(for both intensity and phase). In other cases, the initial guess may bedetermined as an interpolation of the low-resolution intensitymeasurement with a random phase. An example of an initial guess is φ=0and I_(k,l) of any low-resolution image of the sample area. Anotherexample of an initial guess is a constant value. The Fourier transformof the initial guess can be a broad spectrum in the Fourier domain.

At step 2500(1) . . . step 2500(T), the processor reconstructs ahigher-resolution image of each tile (1 to T) independently usingparallel computing. The processor reconstructs the higher-resolutionimage of each tile by iteratively combining low-resolution intensityimages in Fourier space. The recovery process described with respect toFIG. 18, 19, or 23 can be used.

At step 2590, the processor combines the higher-resolution tile imagesinto a full field-of view higher-resolution image. In some cases,combining tile images comprises an imaging-blending process such as, forexample, alpha blending.

At optional step 2600, the image data of the recovered higher-resolutiontwo-dimensional image of the sample area is displayed on a display(e.g., display 230). In one aspect, the method with tile imaging mayfurther comprise a procedure that accounts for differences in incidentangles between different tiles based on the distance between the tilesand each light element.

C. Refocusing and Auto-Focusing

Conventional high NA microscopes and other imaging devices typicallyhave a limited depth of field. For example, the depth-of-field of aconventional microscope with a 20× objective lens with 0.4 NA is about 5μm. With a conventional microscope, resolution degrades as the samplemoves away from the in-focus plane due to its limited depth-of-field. Toimprove resolution using a conventional microscope, the operatortypically moves the stage to mechanically bring the sample back intofocus. In this regard, a precise mechanical stage is needed to bring asample into the in-focus position with sub-micron accuracy.

In certain aspects, a Fourier ptychographic imaging system can refocusthe sample without mechanically moving the sample. For example, theFourier ptychographic imaging method may comprise steps that refocus anout-of-focus sample during the recovery process. With this refocusingprocedure, the Fourier ptychographic imaging system can expand itsdepth-of focus beyond the physical limitations of its filtering opticalelement. In certain cases, a Fourier ptychographic imaging system may beable auto-focus the sample.

During operation of a Fourier ptychographic imaging system, thez-position of the sample plane may not be known a priori. In certainaspects, a Fourier ptychographic imaging method may include one or moreauto-focusing steps that determines the z-position of the sample planeand uses this z-position to digitally refocus the sample. For example,the a Fourier ptychographic imaging method described with respect toFIG. 19 may further comprise a step during or before step 1520 thatcomputes the z-position of the sample plane. The Fourier ptychographicimaging system may the perform autofocusing by using the processor toperform steps 1520 and 1540 in FIG. 19 using the computed z-position ofthe sample. To compute the z-position of the sample plane, the methodmay determine an auto-focusing index parameter. The auto-focusing indexis defined by the following equation:Auto-focusing index: 1/Σabs(√{square root over (I _(lf))}−√{square rootover (I _(lfm))})  (Eqn. 4)

Where:

-   -   √{square root over (I_(lf))} is the amplitude image from the        low-pass filtering, and    -   √{square root over (I_(lfm))} is the actual low-resolution        measurement

The summation in Eqn. 4 is for all oblique incidence angles. After theFourier ptychographic imaging method computes the estimated z-positionof the sample plane, the Fourier ptychographic imaging method candigitally refocus to the estimated z-position. In some cases, thehigher-resolution image solution has been found to converge better whenusing an accurate z-position.

III. Embedded Pupil Function Recovery (EPRY)

A Fourier ptychographic imaging method can be described as asuper-resolution technique that employs angularly varying illuminationand a phase retrieval algorithm. In some cases, a Fourier ptychographicimaging system employing this method may surpass the diffraction limitof its objective lens. In certain examples described herein, a Fourierptychographic imaging system may be in the form of a Fourierptychographic microscope (FPM).

In certain low NA, large field-of-view (FOV) embodiments of the Fourierptychographic imaging system described herein, the system comprises anoptical element with a low NA. This low NA, large FOV system can employa Fourier ptychographic imaging method that can scale up thespace-bandwidth product (SBP) by more than an order of magnitude. Anexample of a description of SBP can be found in A. Lohmann, R. Dorsch,D. Mendlovic, Z. Zalevsky, and C. Ferreira, “Space-bandwidth product ofoptical signals and systems,” J. Opt. Soc. Am. A13(3), 470-473 (1996),which is hereby incorporated by reference for this SBP description. Inthese low NA, large FOV embodiments, an aberration in the low NAobjective lens or other system aberrations can become the limitingfactor to further increasing SBP of the system.

In certain embodiments, the wavefront correction process described inSection II(A) can be implemented into the Fourier ptychographic imagingmethod to correct for a spatially varying aberration of the objectivelens in the system. In one example discussed in reference to G. Zheng,R. Horstmeyer, & C. Yang, “Wide-field, high-resolution Fourierptychographic microscopy,” Nature Photonics, 7(9), 739-745 (2013), aFourier ptychographic microscope that implements this wavefrontcorrection process was able to produce a high-resolution (e.g., about0.78 um, 0.5 NA), wide-FOV (e.g., about 120 mm²) microscope with a finalSBP of about 1 gigapixel microscope. This Fourier ptychographicmicroscope provides imaging capabilities that may be suited for manybiomedical applications such as digital pathology, haematology andimmunohistochemistry.

Typically, the wavefront correction process described in Section II(A)uses a single pre-characterized spatially varying aberration of theFourier ptychographic imaging system in its process. In many cases, thespatially varying aberration of the Fourier ptychographic imaging systemis measured in a calibration process and used as input into the Fourierptychographic imaging method. Examples of calibration techniques thatcan be used to characterize spatially varying aberrations are describedin G. Zheng, X. Ou, R. Horstmeyer, and C. Yang, “Characterization ofspatially varying aberrations for wide field-of-view microscopy,” Opt.Express 21(13), pp. 15131-15143 (2013), H. Nomura and T. Sato,“Techniques for measuring aberrations in lenses used in photolithographywith printed patterns,” Appl. Opt. 38(13), pp. 2800-2807 (1999), J.Wesner, J. Heil, and Th. Sure, “Reconstructing the pupil function ofmicroscope objectives from the intensity PSF,” in Current Developmentsin Lens Design and Optical Engineering III, R. E. Fischer, W. J. Smith,and R. B. Johnson, eds., Proc. SPIE 4767, pp. 32-43 (2002), all of whichare hereby incorporated by reference for the description of calibrationtechniques for measuring spatially varying aberrations in lenses. Thesedescribed calibration techniques are typically computationally onerousand sensitive to the movement of components in the system or otherchanges to the system. For example, any movement of the objective lensor a switch of the imaging camera (e.g., radiation detector) can changethe spatially varying aberration and require re-characterization of theaberration.

Some conventional adaptive wavefront correction processes can makewavefront corrections adaptively with aberration measurements takenperiodically over time. An example of an adaptive wavefront correctionprocess is described in Z. Bian, S. Dong, and G. Zheng, “Adaptive systemcorrection for robust Fourier ptychographic imaging,” Opt. Express21(26), 32400-32410 (2013). However, this example and other conventionaladaptive wavefront correction processes typically involve a globaloptimization technique that is computationally expensive. The heavycomputational burden of this process may limit the orders of aberrationsthat can be corrected within the time restrictions of the imagingprocess and/or of the computational resources.

In certain embodiments, a Fourier ptychographic imaging system may beconfigured to employ a Fourier ptychographic imaging method implementingan embedded pupil function recovery (EPRY) technique that does notrequire a priori knowledge of the aberration and does not use globaloptimization. Instead, the EPRY technique can be implemented into theFourier ptychographic imaging method to recover both the Fourierspectrum of the sample and the pupil function of the Fourierptychographic imaging system simultaneously from the captured sequenceof intensity images. In these embodiments, an aberration-free,high-resolution image of the sample can be recovered and the aberrationbehavior of the Fourier ptychographic imaging system can be estimatedfrom the recovered pupil function without the need of a calibrationprocess to pre-characterize the spatially varying aberration. Moreoversince this EPRY technique does not require optimization techniques orother computationally expensive process, a Fourier ptychographic imagingmethod that employs the EPRY technique is more computationally efficientthan conventional adaptive wavefront correction. Furthermore, an imagingsystem that uses a Fourier ptychographic imaging method that employs theEPRY technique may be able to provide higher quality images thanconventional adaptive wavefront correction systems since it can moreefficiently (less time) process images based on greater numbers of loworder aberrations.

The approach used by the Fourier ptychographic imaging method andconventional ptychography differ in many respects. In conventionalptychography, probe illumination is spatially panned across the samplewhile the far field diffraction patterns are imaged and recorded. Inconventional ptychography, the phase retrieval methods rely on accuratecharacterization of the probe function. In conventional ptychography,the inaccuracies of the characterization may be based on the features ofthe aperture (or focusing optics) that generates the illuminating beam.In contrast, Fourier ptychographic imaging uses a different illuminationapproach to provide oblique plane wave illumination at differentincidence angles, resulting in a sequence of shift version of the sampleFourier spectrum. In Fourier ptychographic imaging, the phase retrievalmethods rely on accurate characterization of the pupil function of theoptical system. In Fourier ptychographic imaging, the inaccuracies ofthe characterization may be based on the aberration of the imagingsystem.

Implementing the EPRY Technique into the Fourier Ptychographic ImagingMethod

In certain embodiments, a Fourier ptychographic imaging system employs aFourier ptychographic imaging method that implements the EPRY technique.The Fourier ptychographic imaging method generally comprises anacquisition process, an image reconstruction process, and an optionaldisplay process. The image acquisition process comprises illuminatingthe sample with oblique plane waves from N varying incidence angles(θx_(i,j), θy_(i,j)) and capturing a sequence of M images. Theacquisition process may be generally expressed as a complexmultiplication e(r)=s(r) exp(iU_(M)·r) where s(r) is the exit light wavefrom a thin sample, which is illuminated by the oblique plane wave witha wavevector and where r=(x, y) is the coordinate in the spatial domainand u=(k_(x), k_(y)) is the coordinate in the spatial frequency domain(Fourier domain). The light wave that propagates to the radiationdetector is the convolution of the exit wave and the spatially invariantpoint spread function p (r) of the Fourier ptychographic imaging systemwhere the intensity is recorded, i.e. I_(U) _(M) =|e(r)

p(r)|². In the Fourier domain:I _(U) _(M) =|F ⁻¹ {F[e(r)]*F[p(r)]}|² =|F ⁻¹ {S(u−U_(M))*P(u)}|²  (Eqn. 5)

-   -   Where:        -   S(u)=F{s(r)} is the Fourier spectrum of the sample, and        -   P (u)=F{p(r)} is the pupil function of the image system            In some cases, the image reconstruction process recovers            S(u) and P(u) that satisfy Eqn. 5 for all M measured            intensity images. In Section III, P(u) refers to the pupil            function distribution, S(u) refers to the sample Fourier            spectrum distribution (also referred to as Ĩ_(h) in previous            sections), s(r) refers to a sample spatial distribution, M            refers to the number of captured images, B refers to the            total number of outer loops executed, a refers to the inner            loop index variable and b refers to the outer loop index            variable.

In some Fourier ptychographic imaging methods described in previoussections (e.g., an example of the method described with respect toflowchart in FIG. 18), a pupil function is initially estimated from thepre-characterized aberration. If the estimated pupil estimation isprecise, the iterative process determines an S(u) that satisfies Eqn. 5.Since the image reconstruction process in these iterations only renewsthe sample spectrum while keeping the pupil function unchanged, animprecise estimated pupil function will result in a poor recovery.Inaccuracy in the estimated pupil function can be caused, for example,by limited orders of aberration considered in the pre-characterizationcalibration process or by mechanical or optical changes in the imagingsystem.

In embodiments of a Fourier ptychographic imaging system that employ amethod that implements the EPRY technique, a pre-characterizedaberration is not required. Instead, the EPRY technique can recover boththe Fourier spectrum of the sample and the pupil function of the systemsimultaneously during its iterative process. The method is generallydescribed with respect to the flowchart shown in FIG. 17. The details ofthe steps of the acquisition portion (steps 1100, 1200, and 1300) andthe optional display portion (step 1600) are similar to those describedwith respect to the flowchart shown in FIG. 17. The image acquisitionprocess comprises illuminating the sample with oblique plane waves fromN varying incidence angles (θx_(i,j), θy_(i,j)) and capturing a sequenceof M images. Details of steps 1400 and 1500 of the reconstructionprocess are described in detail with respect to the flowchart shown inFIG. 23. FIG. 23 is a flowchart depicting details of the steps 1400 and1500 that implement the EPRY technique into the Fourier ptychographicimaging method, according to an embodiment.

At step 1400, the sample spectrum and pupil function are initialized asS₀(u) and P₀(u) respectively. In addition, the outer loop indexvariable, b, is set to 1 (first iteration) and the inner loop indexvariable, a, is set to 0. Outer loop index variable, b is the indexincrementing the reconstruction process iterations and inner loop indexvariable, a, is the index incrementing the incidence angle. In thecycles of the inner loop, M captured images are addressed in thesequence: I_(U) _(a) (r), a=0 to M−1, where M is the number of capturedimages, and each is considered in turn, with both the pupil function andsample spectrum updated at each loop.

In one embodiment, the initial sample spectrum S₀(u) may be determinedby first initialized a sample image in the spatial domain, and thenapplying a Fourier transform to obtain an initialized sample spectrum inthe Fourier domain. In some cases, the initial guess may be determinedas a random complex matrix (for both intensity and phase). In othercases, the initial guess may be determined as an interpolation of thelow-resolution intensity measurement with a random phase. An example ofan initial guess for S₀(u) may be interpolated from one of the capturedintensity images. Another example of an initial guess is a constantvalue. The Fourier transform of the initial guess can be a broadspectrum in the Fourier domain.

In some embodiments, the initial pupil function guess P₀(u) may be acircular shaped low-pass filter, with all ones inside the pass band,zeros out of the pass band and uniform zero phase. In one example, theradius of the pass band is NA×2π/λ, where NA is the numerical apertureof the filtering optical element (e.g., objective lens) and λ is theillumination wavelength. An example of an initial pupil function guesswould be based on assuming the system is aberration free, phase=0.

At step 3010, it is determined whether b=1 i.e. it is the firstiteration of the outer loop. If it is determined that it is not thefirst iteration, then the initial pupil function and the sample spectrumin the Fourier domain are set to the data determined in the last cycleof the inner loop: S₀(u)=S_(M-1)(u) and P₀(u)=P_(M-1)(u) at step 3020.If it is determined that it is the first iteration, then the methodproceeds to step 3030.

In the a^(th) cycle of the inner loop, with the knowledge of thereconstructed S_(a)(u) and P_(a)(u) from the previous cycle of the innerloop, the exit wave at the pupil plane while the sample is illuminatedby a wavevector U_(n) can be simulated using:φ_(a)(u)=P_(a)(u)S_(a)(u−U_(n)) with the S_(a)(u) and P_(a)(u) from theprevious cycle. At step 3030, the processor shifts the sample spectrumaccording to the illumination angle and multiplies by the pupil functionaccording to: φ_(a)(u)=P_(a)(u)S_(a)(u−U_(n)). The pupil functioncomprises both an amplitude and a phase factor. The phase factor of thepupil function is generally associated with defocus or other aberrationassociated with the optical system. The amplitude of the pupil functionis usually associated with the objective lens aperture shape of theoptical system. By multiplying the sample spectrum by the pupil functionin the Fourier domain, the processor both filters the higher-resolutionsolution by multiplying by the modulus (computed amplitude component) ofthe pupil function and also multiplies by the phase factor of the pupilfunction. Multiplying the sample spectrum by the modulus filters thehigher-resolution image in the Fourier domain for a particular planewave incidence angle (θ_(x) ^(a), θ_(y) ^(a)) with a wave vectorU_(a)=(k_(x), k_(y)). An image captured with illumination U_(a) based onthe a^(th) illumination incidence angle is referred to in this sectionas I_(Ua)(r). By multiplying the sample spectrum by the modulus, theprocessor filters a region from the sample spectrum S(u) in the Fourierdomain. In cases with a filtering optical element in the form of anobjective lens, this region takes the form of a circular pupil aperturewith a radius of NA*k₀, where k₀ equals 2π/λ (the wave number invacuum), given by the coherent transfer function of an objective lens.The center of the circular region in Fourier space corresponds to theassociated illuminating incidence angle of this a^(th) cycle of theinner loop. For an oblique plane wave incidence with a wave vectorU_(a)=(k_(x), k_(y)), the region is centered about a position (k_(x),k_(y)) in the Fourier domain.

At step 3040, the processor takes the inverse Fourier transform asfollows: φ_(a)(r)=F⁻¹{φ_(a)(u)}. At step 3050, the processor imposes anintensity constraint. In this step 3050, the modulus (computed amplitudecomponent) of the simulated region in Fourier space is replaced with thelow resolution intensity measurement I_(U) _(a) (r) captured by theradiation detector associated with an illumination wavevector U_(a). Thecomputed amplitude component is replaced by the square-root of the realintensity measurement I_(Ua)(r) according to:

${\phi_{a}^{\prime}(r)} = {\sqrt{I_{U_{a}}(r)}{\frac{\phi_{a}(r)}{{\phi_{a}(r)}}.}}$This forms an updated lower resolution image.

At step 3060, a Fourier transform is applied to the updated lowerresolution image. In this step, an updated exit wave is calculated via aFourier transform according to: φ′_(a)(u)=ℑ{φ′_(a)(r)}.

At step 3070, the processor refreshes the Fourier spectrum guess of thehigher resolution solution by updating the exit wave data and replacingdata in a corresponding region of the Fourier domain as the updated exitwave data associated with incidence wave vector U_(n)=(k_(x), k_(y)).The processor updates the exit wave data using a sample spectrum updatefunction. An example of a sample spectrum update function is given by:

$\begin{matrix}{{S_{a + 1}(u)} = {{S_{a}(u)} + {\alpha{\frac{P_{a}^{*}\left( {u + U_{a}} \right)}{{{P_{a}\left( {u + U_{a}} \right)}}_{\max}^{2}}\left\lbrack {{\phi_{a}^{\prime}\left( {u + U_{a}} \right)} - {\phi_{a}\left( {u + U_{a}} \right)}} \right\rbrack}}}} & \left( {{Eqn}.\mspace{14mu} 6} \right)\end{matrix}$By using such a spectrum update function, the updated value of thesample spectrum may be extracted from the difference of the two exitwaves by dividing out the current pupil function. By multiplying withthe conjugates using Eqn. 6 and Eqn. 7, the sample spectrum can beseparated from the pupil function so that the sample spectrum can berefreshed separately from the pupil function. In some cases, acorrection is added to the sample spectrum guess with weightproportional to the intensity of the current pupil function estimate.The constant α adjusts the step size of the update. In one example, α=1.During the cycles of the inner loop, the data is updated as overlappingregions in the Fourier domain.

Concurrently with step 3070, at step 3080 the processor refreshes theguess of the pupil function in the Fourier domain as: P_(a+1)(u). Anexample of a pupil update function that can be used here is given by:

$\begin{matrix}{{P_{a + 1}(u)} = {{{SP}_{a}(u)} + {\beta{\frac{S_{a}^{*}\left( {u + U_{a}} \right)}{{{S_{a}\left( {u + U_{a}} \right)}}_{\max}^{2}}\left\lbrack {{\phi_{a}^{\prime}(u)} - {\phi_{a}(u)}} \right\rbrack}}}} & \left( {{Eqn}.\mspace{14mu} 7} \right)\end{matrix}$The constant β adjusts the step size of the pupil function update andβ=1 is used in this paper. Using this pupil update function, thecorrection of the pupil function is extracted from the difference of thetwo exit waves by dividing out the current sample spectrum estimate, andadded to the current pupil function guess with weight proportional tothe intensity of the current sample spectrum estimate. By multiplying bythe conjugate using Eqn. 7, the pupil function can be separated from thesample spectrum and refreshed separately.

At step 3082, the processor imposes a pupil function constraint on theupdated pupil function. Imposing the pupil function constraint maysuppress noise. In the example of a microscope system, a physicalcircular aperture stop may be set to define the NA, thus the area in thepupil function that corresponds to the stop should always be zero. Thenon-zero points in the updated pupil function in the regioncorresponding to the stop are caused by the noise in image acquisition,and are set to zero to eliminate the noise.

The inner loop of the method continues to cycle until all M capturedimages in the sequence I_(U) _(a) (r) are used to update the pupil andsample spectra, at which point an iteration of the outer loop iscomplete. The cycles run from a=0 to M−1. At step 3090, the processordetermines whether a=M−1. If the processor determines that a does notequal M−1, then not all the M captured images have been used. In thiscase, the loop index a will be incremented at step 3092, and the methodwill return to step 3030 based on the next captured image associatedwith another incidence angle.

If the processor determines that a does equal M−1, the method continuesto step 3094. If the processor determines that a does not equal M−1, themethod continues to step 3092. At step 3092, the outer loop index isincremented a=a+1 to the next incidence angle. The method will thenreturn to start a new cycle at step 3030.

At step 3094, the processor determines whether b=B. If the processordetermines that b does not equal B, the loop index b will be incrementedat step 3096 to b=b+1 and the loop index a will be reset to 0. Themethod will then return to start a new iteration at step 3010.

If the processor determines that b does equal B, then the iterationsstop and the method continues to step 3098. At step 3098, the inverseFourier transformed back to the spatial domain to generate a highresolution, modulus and phase distribution of the sample. A highresolution image of the sample can be generated from this data. Themethod then returns to optional step 1600 in FIG. 17 to send data todisplay a high resolution modulus and phase image of the image on adisplay (e.g., display 230 in FIG. 1).

Fourier Ptychographic Imaging Method Implementing EPRY TechniqueCompared to Fourier Ptychographic Imaging Method without AberrationCorrection

FIG. 24 illustrates nine images, the first row of images, 3101, 3102,and 3103, corresponds to datasets used as the modulus and phase of thesample and pupil function phase for simulation, the second row ofimages, 3104, 3105, and 3106, corresponds image data resulting from Biterations of a Fourier ptychographic imaging method that does not useaberration correction, and the third row of images, 3107, 3108, and3109, corresponds to image data resulting from B iterations of a Fourierptychographic imaging method that implements the EPRY technique,according to embodiments. The first column corresponds to intensity, thesecond column corresponds to phase, and the third column corresponds tothe pupil function phase. The first two images of the first row, 3101and 3102, are the ground truth sample modulus (intensity) and phase. Thethird image of the first row, 3103, is the pupil function phase. Thesimulated dataset is based on a simulated microscope system having anNA=0.08 with a wavefront aberration, resulting in a circularly shapedpupil function with a radius of 13 pixels and a pupil function phase asshown in a3. Each of the images 3101 and 3102 contains 512*512 pixelswith pixel size 0.2 um.

The same initial guess of the pupil phase is used in both the Fourierptychographic imaging method with and without implementing the EPRYtechnique. The pupil function has a circular shape with a low passfilter size which is the same size as the phase circle. In these runs,225 images are measured with different plane wave illuminations withoverlap in Fourier domain of about 70%. In both runs, the initial guessof the pupil function is set as a circular shape low-pass filter radiusof 13 pixels with zero phase, as shown in 3106 of FIG. 24, and the firstimage in the sequence is up-sampled and Fourier transformed to serve asthe initial guess of sample spectrum. Both methods were run for 100iterations and the results are shown in 3104 and 3105 and 3107, 3108,and 3109 in FIG. 24.

The image reconstruction from the method without aberration correctionresults in the image data in 3104 and 3105. The results of imagereconstruction from the method with the EPRY technique are in 3107,3108, and 3109. In this example, the results with EPRY provide improvedquality results, which is because the aberated wavefront of the pupilfunction repeatedly influenced the low and high frequency components ofthe sample spectrum. In addition, there may be a significant degree ofcrosstalk between the modulus and phase images resulting from the lackof knowledge about the pupil function phase distribution. In contrast,the method implementing the EPRY technique separates the pupil functionfrom the sample spectrum, resulting in an improved quality image and anaccurate measurement of the real pupil function phase. Because theilluminations do not cover the entire Fourier spectrum of the sample,there exists a small amount of crosstalk in the modulus and phase image,and also, resulting in several phase-wrapped pixels in the reconstructedpupil function.

One advantage of implementing the EPRY technique may be improvedconvergence and consequently fewer iterations and computationalresources required. Convergence may be a measure of the normalized meansquare error metrics in each iteration given by:

$\begin{matrix}{{{E^{2}(m)} = \frac{\sum\limits_{u}^{\;}\;{{{S(u)} + {\alpha\;{S_{b}(u)}}}}^{2}}{\sum\limits_{u}^{\;}\;{{S(u)}}^{2}}}{{{Where}\mspace{14mu}{parameter}\mspace{14mu}\alpha} = \frac{\sum\limits_{u}^{\;}\;{{S(u)}S_{b}^{*}}}{\sum\limits_{u}^{\;}\;{{S_{a}(u)}}^{2}}}} & \left( {{Eqn}.\mspace{14mu} 8} \right)\end{matrix}$

Here, S(u) refers to the actual sample spectrum distribution andS_(b)(u) refers to the reconstructed sample distribution after biterations. FIG. 25 is a plot of mean square error E² (b) vs. iterationsto convergence for runs of a Fourier ptychographic imaging method thatimplements the EPRY technique and one that does not implement the EPRYtechnique, according to embodiments. In these runs, the E²(b) iscalculated over the center 128×128 pixel area. For the reconstructedsample spectrum, the method implementing EPRY technique has asignificantly faster convergence rate compared to the method that doesnot use EPRY. The method that does not implement the EPRY techniqueresults in an error of less than 0.01. The method that does notimplement the EPRY technique, which does not iteratively correct thepupil function, and the error in the sample spectrum does not decreasebelow 0.08 after 20 iterations, which is the limit imposed by theuncorrected pupil function. As we can see in the plot, the convergenceof the reconstructed pupil function using the EPRY technique convergesslower than the sample spectrum at the first few iterations, then thefinal result has an error of about 0.05.

Comparison of Method with EPRY Technique to a Method without AberrationCorrection on Experimental Data

In some cases, implementing the EPRY technique may not only improveimage quality output from the Fourier ptychographic imaging system andmake its method more computationally efficient, but also the pupilfunction can be recovered and used to characterize the spatially varyingaberration in the system. Section V describes an example of a Fourierptychographic imaging system implementing the EPRY technique andconfigured to characterize the spatially varying aberration of itsoptical system. In one case, such a system can then be configured toadaptively correct for the spatially varying aberration determined ateach iteration.

In one example, Fourier ptychographic imaging system may comprise anobjective lens (2×, NA=0.08 objective) and specimen receptacle of aconventional microscope, a variable illuminator such as a programmablecolor LED matrix, a radiation detector, and a computing device incommunication with the radiation detector and/or LED matrix. Theradiation detector may be, for example, a CCD camera as a radiationdetector mounted on top of the objective lens. The setup may be similarto the one shown in FIGS. 4 and 5.

FIG. 26 are six (6) images of reconstructed image data generated by sucha Fourier ptychographic imaging system that employs a Fourierptychographic imaging method without aberration correction and a Fourierptychographic imaging method implementing the EPRY technique, accordingto embodiments. Each of the methods used a blood smear dataset. In bothcases, the initial guess of the pupil function was a circular shape lowpass filter, whose radius was determined by the NA, with no phase, andthe reconstructed region is located 35% from the center of the field ofview, and the first image was up-sampled and Fourier transformed toserve as the initial guess of sample spectrum. In both cases, 225intensity images were captured using 15 by 15 red LEDs. In both cases,an area of 150 um×150 um of the sample was analyzed from the sample,located at 35% to the edge from the center of the FOV of the imagingsystem, where the aberration is non-negligible.

The first column images, 3110 and 3113, are the reconstructed sampleintensity and phase using the Fourier ptychographic imaging methodwithout aberration correction. The second column images, 3111 and 3114,are the reconstructed sample intensity and phase using the Fourierptychographic imaging method implementing the EPRY technique. The thirdcolumn images, 3112 and 3115, are reconstructed pupil function modulusan phase using the Fourier ptychographic imaging method implementing theEPRY technique.

Images 3110 and 3113 show the intensity and phase distribution of theblood smear using the Fourier ptychographic imaging method withoutaberration correction. In this example, the image is relatively blurrydue to objective aberration at that location in the field of view. Asshown, the contour of the blood cells is not clearly recognizable whichmakes it difficult to distinguish white blood cells from red bloodcells.

In comparison, images 3111 and 3114 based on a method implementing theEPRY technique show a relatively higher quality image. In this example,the morphology of blood cells is clear, the zone of central pallor forthe red blood cells is obvious, and the shape of the nucleus of thewhite blood cell is recognizable. From the phase image 3114, we can alsosee the donut shape of the red blood cell. The pupil function for thisfield of view is also recovered by implementing the EPRY technique andthe recovered pupil function data is illustrated in images 3112 and3115.

The recovered pupil function can be used to determine the properties ofthe optical system. For one example, the size and shape of the modulusof the pupil function reflects the shape and position of the physicalaperture stop. In this case, the modulus part of the pupil functionapproximately remains the same as the initial guess, meaning that thenumerical aperture is well defined by a circular shape aperture. Also,since the pupil function should be centered, but is showing a slightshift of the pupil function to the bottom right, which indicates animprecise estimation of the wavevector U_(n) that is caused by the shiftof the LED matrix from the originally aligned position. In this case,the system implementing the EPRY technique can be further configured tocorrect for this aberration.

As another example, the phase of the pupil function represents thewavefront aberration. A decomposition of the pupil function phasecomponent in Zernike polynomials shows that the coefficient of eachZernike polynomial represents the extent of aberration corresponding tothis Zernike polynomial. In the example system described above, thedecomposition is executed and the coefficients of the first 30 Zernikepolynomials are shown in FIG. 18. Different Zernike polynomialsrepresent different types of aberration, from low order to high orderaccording to the mode number. Mode number 1 represents the piston term,which will cause a constant phase shift to the entire aperture and isnot considered as an aberration. The three dominant modes for thewavefront aberration are mode number 4, 5 and 6, which represent defocusaberration, astigmatism in the x direction and astigmatism in the ydirection respectively. Coma aberration (mode 7 and 8) is not severe forthis FOV but there are some higher order aberrations that arenon-negligible for this position, such as mode 9 (trefoil) and mode 13.

Variations of Recovered Pupil Function Across Field of View andSpectrally

In imaging systems with large field of view, aberration and, byextension, pupil function typically exhibit spatial variations acrossthe field of view and also vary spectrally.

In certain embodiments, a Fourier ptychographic imaging methodimplementing an EPRY technique can use steps similar to those describedin the flowchart shown in FIG. 22 except that steps 1400 and 1500described with reference to FIG. 23 replace steps 2400 and 2500. Byusing this tile approach, the entire field of view can be segmented intosmaller tiles. In one example, the Fourier ptychographic imaging systemhas a 6 mm radius field of view and the entire area is segmented intotiles sized 350 um×350 um. To ensure the effectiveness of the EPRY-FPMalgorithm, the entire field of view was segmented into small tiles wherein each tile the aberration was considered as constant. A Fourierptychographic imaging method implementing an EPRY technique as describedin FIG. 23 is used to reconstruct an image for each tile independently.The reconstructed high resolution, aberration eliminated images for thetiles are combined together to form a full FOV high resolution image.

In certain aspects, the method may assume that the pupil function variescontinuously. In these cases, the reconstructed pupil function from theadjacent tile is used as the initial pupil function guess (instead of aflat phase initial guess) for the current tile to increase theconvergence rate of the iterations in the Fourier ptychographic imagingmethod.

In one embodiment, a Fourier ptychographic imaging system employing amethod implementing the EPRY technique with the tile approach and usingthe reconstructed pupil function from the adjacent tile is used as theinitial pupil function guess for the current tile is used. FIG. 28includes the reconstructed sample image and wavefront aberration of fiveregions over the field of view resulting from a run of this system,according to the embodiments. FIG. 28 includes a full high resolutionmonochrome image reconstruction of blood smear as a center image. Theentire field of view is segmented into smaller tiles, and the aberrationis treated as constant for each tile. A method implementing the EPRYtechnique is run on each time and the reconstructed high resolutionimages are mosaicked together. FIG. 28 also includes five insets showingthe detail of a reconstructed image (left side) and also the wavefrontaberration (right side) at five tile locations across the field of view.As shown, the aberration at the edge is more severe as compared to thecenter. The reconstructed images show a stable image quality from centerto edge resulting from the run of the method implementing the EPRYtechnique.

In a similar embodiment, a Fourier ptychographic imaging systememploying a method implementing the EPRY technique with the approach wasused to render a high resolution, large FOV color image of a pathologyslide shown in FIG. 29. In this example, the variable illuminatorcomprises a LED matrix where the center 15×15 red, green and blue LEDson the LED matrix are lit up individually and 3 intensity distributiondatasets are captured by a CCD camera for each of the red, green, andblue illuminations. For each color channel, the same segmentation andreconstruction processes are executed as described with respect to theexample above. For each tile, because the pupil function which containsthe defocus aberration is separated from the sample spectrum in the EPRYreconstruction process, each color channel is focused at its focalplane. In other words, the axial chromatic aberration, which is causedby different wavelengths focused at different planes, is correctable bythe EPRY technique. Before combining red, green and blue channel imagesin the same tile together, green and blue images are slightly shiftedspatially relative to the red channel to correct for lateral chromaticaberration. The processor can run instructions to determine the correctamount of shift that maximizes the correlation of the red-green imagepair and red-blue image pair respectively. Finally, all the color tilesare mosaicked together for form the full FOV high resolution color imagereconstruction of pathology slide shown in FIG. 29. FIG. 30 includes thereconstructed sample intensity and wavefront aberration of the threeregions. The three channels are combined to generate RGB image. Theimages 3120, 3124, and 3128 are the reconstructed and zoomed in sampleintensity images of three regions in the field of view. The images 3121,3125, and 3129 are the red channel wavefront aberration of the threeregions. The images 3122, 3126, and 3130 are the green channel wavefrontaberration of the three regions. The images 3123, 3127, and 3131 are theblue channel wavefront aberration of the three regions. The differentsizes of circles between the different color channels are caused by thedifferent wavelengths. The shape of the pupil function changed from acircle to an ellipse significantly as moving towards the edge of theimage because the 2× objective being used is not strictly a telecentriclens and, as such, the aperture shape can be expected to changeasymmetrically.

Image Quality Improvement with Fourier Ptychographic Imaging MethodImplementing EPRY Technique

FIG. 31 includes reconstructed images of a USAF target at differentfields of view. The images were reconstructed from three configurationsof the Fourier ptychographic imaging method: 1) a Fourier ptychographicimaging method without aberration correction, 2) a Fourier ptychographicimaging method with a pre-characterized aberration correction, and 3) aFourier ptychographic imaging method implementing EPRY technique withouta pre-characterized aberration, according to embodiments. The USAFtarget is placed at 0%, 27%, 54% and 80% of the entire FOV from thecenter, and four sets of images are captured respectively using the redLED. Images 3140, 3141, 3142, and 3143 are reconstructed by using theFourier ptychographic imaging method without aberration correction.Images 3144, 3145, 3146, and 3147 are reconstructed by using the Fourierptychographic imaging method with a pre-characterized aberrationcorrection. Images 3148, 3149, 3150, and 3151 are reconstructed by usingthe Fourier ptychographic imaging method implementing EPRY techniqueaberration without pre-characterized aberration.

By comparing the images, image quality is shown to degrade due to theaberration in 3142 and 3143 at the 54% and 80% FOV locations. In images3146 and 3147, it is shown that after aberration correction of defocusand astigmatism, the line on Group 9 (line width <1 μm) can be vaguelyresolved at 3146 and 3147. The method using aberration correction with apre-characterized aberration correction typically does not correct forhigher orders of wavefront aberration and only accounts for lowerorders. The reason that higher order aberration were not included isthis method is that the higher order aberration information may beoverwhelmed by the noise of the imaging system using this method ofmeasurement, resulting in imprecise measurement of the higher orderaberration. In addition, these aberrations are highly sensitive tomechanical or optical system drifts.

In contrast, a Fourier ptychographic imaging method implementing theEPRY technique can characterize the entire or nearly entire pupilfunction including the higher orders of wavefront aberration. In theseembodiments, the method can improve image quality since it accounts forhigher orders of wavefront aberration. In these cases, the method may bemore computationally efficient than conventional methods and have a morerobust aberration characterization of the involved lens system.

A Fourier ptychographic imaging system employing a Fourier ptychographicimaging method implementing an EPRY technique can recover both theexpanded sample spectrum and the pupil function of the imaging systemusing the intensity images captured by the radiation detector. Theimplementation of EPRY technique may improve image quality due to thefact that the entangled sample spectrum and pupil function are isolatedfrom the captured images during the recovery process. Moreover, therecovered pupil function which contains wavefront aberration informationof the system can be used to characterize the behavior of the lenses ofthe system. In some cases, the Fourier ptychographic imaging method withthe EPRY technique can be employed to characterize optical systemaberrations. For example, it can be used to benchmark the quality ofimaging systems for comparison purposes. Alternately, the recoveredsystem aberration data can be used to design appropriate correctionoptics as discussed in Section V. Examples of some details of Fourierptychographic imaging are described in Ou, Xiaoze, Guoan Zheng, andChanghuei Yang, “Embedded pupil function recovery for Fourierptychographic microscopy” Optics Express 22, no. 5 (2014): 4960-4972 andG. Zheng, R. Horstmeyer and C. Yang, “Wide-field, high-resolutionFourier ptychographic microscopy,” Nature Photonics, 2013, which arehereby incorporated by reference in their entirety.

V. Sharp Focus Generation Via EPRY and Adaptive Optics

Some conventional microscope systems include a highly focused beam suchas, for example, a confocal microscope, a second-harmonic imagingmicroscope, optical tweezers, etc. A high numerical aperture objectiveis typically used to generate the finely focused light spot in thesesystems. Due to the aberration caused by technological limitations onmaterial and lens design, the focused light spot size in these systemscannot reach the diffraction limit.

In some embodiments, a Fourier ptychographic imaging system employing amethod that implements the EPRY technique can be used to simultaneouslyrecover the extended complex Fourier spectrum of the sample and thepupil function of the system. This system can use the recovered pupilfunction can be used to characterize the wavefront aberration of theoptical system including, for example, an objective lens. In some cases,the Fourier ptychographic imaging system comprises adaptive opticsconfigured to modulate the incident wavefront and correct for thespatially varying aberration in the optical system to generate anaberration free, diffraction-limited focused light spot.

FIGS. 32 and 33 are side views of a Fourier ptychographic imaging system3330 that includes adaptive optics and employs an operational methodthat implements the EPRY technique, according to an embodiment. TheFourier ptychographic imaging system 3330 comprises an optical systemcomprising an objective lens 3310, a beam splitter 3320, and a tube lens3330. The Fourier ptychographic imaging system 3330 also comprises aradiation detector 3340, for example, a CCD camera. The Fourierptychographic imaging system 3330 also comprises a variable illuminator3350, for example, a square LED array having discrete light elements. Inthis example, the variable illuminator 3350 is a square array of 8×8light discrete light elements. In addition, the Fourier ptychographicimaging system 3330 comprises a wavefront modulator 3390 configured tomodulate an incident wavefront to adaptively correct for the spatiallyvarying aberration in the optical system determined during operation.The beam-splitter 3320 is configured to reflect illumination incidentfrom a 45 degree angle from the wavefront modulator 3390 to theobjective lens 3100 and transmits light incident from a 90 degree anglefrom the objective lens 3310.

In FIG. 32, one of the discrete light elements is illuminated andproviding an illumination beam 3352 to a focused spot on the sampleplane 3360. During an image acquisition process, a sample being imagedis placed at the sample plane 3360. In FIG. 33, the wavefront modulator3390 is modulated an incident wavefront.

In a first step of an exemplary operation, the Fourier ptychographicimaging system characterizes the aberration of the objective lens of theoptical system. In this first step, the radiation detector 3340 acquiresa sequence of images associated with different illumination incidenceangles. During a recovery process, the Fourier ptychographic imagingsystem uses a method implementing the EPRY technique to recover thepupil function of the system and the associated wavefront aberration. Ina second step of the exemplary operation, the wavefront aberrationinformation is used to modulate the incident wavefront and generateaberration free, diffraction-limited focused light spot at the sampleplane. In this second step, the wavefront aberration information iscommunicated to the wavefront modulator 3390. The wavefront modulator3390 uses the wavefront aberration information to modulate the incidentplane wave. Because the aberration of the objective lens is compromisedby the modulated wave front, an aberration free, diffraction-limitedfocused light spot can be generated at the sample plane.

In some embodiments, components of a Fourier ptychographic imagingsystem may be configured to replace existing components or addcomponents to an existing imaging system in order to characterize theaberration of the existing optical system from the pupil function andadaptively correct for its aberration. For example, a square array LEDmay replace the original light source of an existing microscope systemto provide angularly varying illumination. A CCD cameral can be mountedto the tube lens to acquire images. A beam splitter and/or wavefrontmodulator can be added that can adaptively correct for the aberrationcharacterized by the system. Components of a computing device such as aprocessor may also be added to the system to perform certain processsteps.

The schematic of the process is shown in FIGS. 32 and 33. In the firststep, a square array LED is used to substitute the original light sourceof the microscope system to provide angularly varying illumination. Asequence of images is captured following the FPM principle, and EPRYalgorithm is implemented to recover the pupil function of the microscopesystem. In the second step, the wavefront aberration information is sentto the wavefront modulator to modulate the incident plane wave. Someexamples of suitable wavefront modulators that can be used include adigital micromirror device, liquid crystal spatial light modulator, andthe like. Because the aberration of the objective lens is compromised bythe modulated wave front, an aberration free, diffraction-limitedfocused light spot can be generated at the sample plane.

VI. Subsystems

FIG. 34 is a block diagram of subsystems that may be present in certainFourier ptychographic imaging system described herein. For example, aFourier ptychographic imaging system may include a processor. Theprocessor may be a component of the Fourier ptychographic imaging systemin some cases. The processor may be a component of the radiationdetector in some cases.

The various components previously described in the Figures may operateusing one or more of the subsystems to facilitate the functionsdescribed herein. Any of the components in the Figures may use anysuitable number of subsystems to facilitate the functions describedherein. Examples of such subsystems and/or components are shown in aFIG. 34. The subsystems shown in FIG. 34 are interconnected via a systembus 4425. Additional subsystems such as a printer 4430, keyboard 4432,fixed disk 4434 (or other memory comprising computer readable media),display 230, which is coupled to display adapter 4438, and others areshown. Peripherals and input/output (I/O) devices, which couple to I/Ocontroller 4440, can be connected by any number of means known in theart, such as serial port 4442. For example, serial port 4442 or externalinterface 4444 can be used to connect the computing device 200 to a widearea network such as the Internet, a mouse input device, or a scanner.The interconnection via system bus 4425 allows the processor tocommunicate with each subsystem and to control the execution ofinstructions from system memory 4446 or the fixed disk 4434, as well asthe exchange of information between subsystems. The system memory 4446and/or the fixed disk 4434 may embody the CRM 220 in some cases. Any ofthese elements may be present in the previously described features.

In some embodiments, an output device such as the printer 4430 ordisplay 230 of the aperture scanning Fourier ptychographic system canoutput various forms of data. For example, the aperture scanning Fourierptychographic system can output 2D color/monochromatic images (intensityand/or phase), data associated with these images, or other dataassociated with analyses performed by the aperture scanning Fourierptychographic system.

Modifications, additions, or omissions may be made to any of theabove-described embodiments without departing from the scope of thedisclosure. Any of the embodiments described above may include more,fewer, or other features without departing from the scope of thedisclosure. Additionally, the steps of the described features may beperformed in any suitable order without departing from the scope of thedisclosure.

It should be understood that certain features of embodiments of thedisclosure described above can be implemented in the form of controllogic using computer software in a modular or integrated manner. Basedon the disclosure and teachings provided herein, a person of ordinaryskill in the art will know and appreciate other ways and/or methods toimplement certain features using hardware and a combination of hardwareand software.

Any of the software components or functions described in thisapplication, may be implemented as software code to be executed by aprocessor using any suitable computer language such as, for example,Java, C++ or Perl using, for example, conventional or object-orientedtechniques. The software code may be stored as a series of instructions,or commands on a CRM, such as a random access memory (RAM), a read onlymemory (ROM), a magnetic medium such as a hard-drive or a floppy disk,or an optical medium such as a CD-ROM. Any such CRM may reside on orwithin a single computational apparatus, and may be present on or withindifferent computational apparatuses within a system or network.

Although the foregoing disclosed embodiments have been described in somedetail to facilitate understanding, the described embodiments are to beconsidered illustrative and not limiting. It will be apparent to one ofordinary skill in the art that certain changes and modifications can bepracticed within the scope of the appended claims.

One or more features from any embodiment may be combined with one ormore features of any other embodiment without departing from the scopeof the disclosure. Further, modifications, additions, or omissions maybe made to any embodiment without departing from the scope of thedisclosure. The components of any embodiment may be integrated orseparated according to particular needs without departing from the scopeof the disclosure.

What is claimed is:
 1. A Fourier ptychographic imaging system employingembedded pupil function recovery, comprising: a variable illuminatorconfigured to illuminate a sample at a plurality of oblique illuminationincidence angles; an objective lens configured to filter light issuingfrom the sample based on its numerical aperture; a radiation detectorconfigured to receive light filtered by the objective lens and capture aplurality of intensity images corresponding to the plurality of obliqueillumination incidence angles; and a processor configured to iterativelyand simultaneously update a pupil function and a separate samplespectrum, wherein the sample spectrum is updated iteratively for eachillumination incidence angle at overlapping regions in the Fourierdomain with Fourier transformed intensity image data, wherein theoverlapping regions correspond to the plurality of illuminationincidence angles and the numerical aperture of the objective lens. 2.The Fourier ptychographic imaging system of claim 1, wherein theprocessor further configured to inverse transform the updated samplespectrum to determine an image of the sample, wherein the image has ahigher resolution than the captured intensity images.
 3. The Fourierptychographic imaging system of claim 1, wherein the processor isfurther configured to determine an aberration from the updated pupilfunction; and further comprising a wavefront modulator configured toadaptively correct an incident wavefront based on the determinedaberration.
 4. The Fourier ptychographic imaging system of claim 1,wherein the objective lens has a numerical aperture between about 0.02and 0.13.
 5. The Fourier ptychographic imaging system of claim 1,wherein the objective lens has a numerical aperture of about 0.08. 6.The Fourier ptychographic imaging system of claim 1, wherein thevariable illuminator comprises a plurality of discrete light elements,wherein the plurality of oblique illumination incidence angles areassociated with different discrete light elements.
 7. The Fourierptychographic imaging system of claim 1, wherein the variableilluminator comprises a circular array of discrete light elements. 8.The Fourier ptychographic imaging system of claim 1, wherein theoverlapping regions overlap by between 20% and 90% in area.
 9. TheFourier ptychographic imaging system of claim 1, wherein the overlappingregions overlap by between 2% and 99.5% in area.
 10. The Fourierptychographic imaging system of claim 1, wherein the overlapping regionsoverlap by about 66% in area.
 11. A method of Fourier ptychographicimaging employing embedded pupil function recovery, the methodcomprising: illuminating a sample from a plurality of incidence anglesusing a variable illuminator; filtering light issuing from the sampleusing an optical element; capturing a plurality of variably-illuminatedintensity image distributions of the sample using a radiation detector;simultaneously updating a pupil function and a separate sample spectrum,wherein the sample spectrum is updated in overlapping regions withFourier transformed variably-illuminated intensity images measurements,wherein the overlapping regions corresponds to the plurality ofincidence angles and the numerical aperture of the lens; and inverseFourier transforming the recovered sample spectrum to recover an imagehaving a higher resolution than the intensity images.
 12. The method ofFourier ptychographic imaging employing embedded pupil function recoveryof claim 11, the method further comprising inverse transforming theupdated sample spectrum to determine an image of the sample, wherein theimage has a higher resolution than the captured intensity images. 13.The method of Fourier ptychographic imaging employing embedded pupilfunction recovery of claim 11, the method further comprising:determining an aberration from the updated pupil function; andadaptively correcting for the determined aberration using a wavefrontmodulator.
 14. The method of Fourier ptychographic imaging employingembedded pupil function recovery of claim 11, wherein the objective lenshas a numerical aperture between about 0.02 and 0.13.
 15. The method ofFourier ptychographic imaging employing embedded pupil function recoveryof claim 11, wherein the objective lens has a numerical aperture ofabout 0.08.
 16. The method of Fourier ptychographic imaging employingembedded pupil function recovery of claim 11, wherein the variableilluminator comprises a circular array of discrete light elements. 17.The method of Fourier ptychographic imaging employing embedded pupilfunction recovery of claim 11, wherein the overlapping regions overlapby between 20% and 90% in area.
 18. The method of Fourier ptychographicimaging employing embedded pupil function recovery of claim 11, whereinthe overlapping regions overlap by between 2% and 99.5% in area.
 19. Themethod of Fourier ptychographic imaging employing embedded pupilfunction recovery of claim 11, wherein the overlapping regions overlapby about 66% in area.